OpenTelemetry Falcon 4.x integration

[vytas7]

OpenTelemetry is a collection of tools, APIs, and SDKs. You use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) for analysis in order to understand your software's performance and behavior.

It's yet to be seen how much traction OpenTelemetry gains, but it seems to be getting embraced by large actors.

OpenTelemetry Python instrumentation actually has a Falcon 2.0 instrumentation plugin!

It would be great to extend this instrumentor for the upcoming Falcon 3.x series.
Some things that I expect to break or be missing:

• falcon.API ➡️ falcon.App renaming might not get picked by the plugin's monkey patching unless one is using the deprectated API alias
• Falcon ASGI support
• The plugin's error heuristics could probably use our insights (do not rely on exception names etc)
• Anything else?

Related/a subset of: #10

[vytas7]

Falcon 3.0 WSGI-only (it seems ASGI is unsupported?) integration might be resolved by open-telemetry/opentelemetry-python-contrib#644 by @adriangb.

[adriangb]

Well we would love your help over there!

Specifically it seems that catching tracebacks is very different in two and three.

I also think it would be great if you could take an overall look at the package, I'm sure you could find improvements 😃

[owais]

Now that we have brought this up, I'd like to point out that a number of well-known projects have added or decided to add support for OpenTelemetry out of the box. I'm one of the maintainers of OpenTelemetry Python and I'd be glad to help Falcon do the same if that is something of interest to Falcon maintainers.

Falcon itself would only take dependency only on opentelemetry-api which basically is a very lightweight (essentially a "noop") package that just provides the OpenTelemetry interfaces. End users can then install opentelemetry-sdk and other components to activate OpenTelemetry at runtime and automatically add observability to their Falcon services.

Some other OSS projects I know of that have decided to support OpenTelemetry officially:

Spring Sleuth: https://github.com/spring-cloud-incubator/spring-cloud-sleuth-otel
Vert.X: https://vertx.io/docs/vertx-opentelemetry/java/
http4k: https://www.http4k.org/guide/reference/opentelemetry/
Apache Camel: https://camel.apache.org/components/3.7.x/others/opentelemetry.html
Clickhouse: https://clickhouse.tech/docs/en/operations/opentelemetry/
Dapr: https://docs.dapr.io/operations/monitoring/tracing/open-telemetry-collector/
Open Distro for Elasticsearch: https://opendistro.github.io/for-elasticsearch/blog/releases/2020/12/announcing-trace-analytics/
Go Redis: https://github.com/go-redis/redis/
Hadoop: https://issues.apache.org/jira/browse/HADOOP-15566
Ambassador: https://app.getambassador.io/initializer/
Envoy: https://blog.envoyproxy.io/envoy-support-for-opentelemetry-access-logging-e4b08160d32c
https://kubernetes.io/blog/2021/09/03/api-server-tracing/
Jenkins: https://plugins.jenkins.io/opentelemetry/
Jaeger: https://medium.com/jaegertracing/jaeger-embraces-opentelemetry-collector-90a545cbc24

[vytas7]

Hi @owais!
That's an option too. Honestly speaking, we'd probably be reluctant to go from 0 hard dependencies to 1 just for this purpose.

OTOH, we could probably make it optional, and host the plugin package under the Falconry umbrella, while shipping some very basic instrumentation affordances in the framework itself (if needed). We could also add an install "extra" dependency pulling in the supported instrumentation plugins.

@owais Do you know any Python frameworks that could serve as an example how to properly ship and integrate it?

@kgriffs @CaselIT thoughts?

[adriangb]

Maybe just adding well designed hooks for telemetry/metrics/logging + fixing issues like exception capture (#1942 as well as open-telemetry/opentelemetry-python-contrib#644) might be enough? Basically making it dead-simple for users (or the OpenTelemetry team) to hook into Falcon. Perhaps even write documentation and have tests around hooking opentelemetry-api, without adding it as a dependency (only a test dependency).

[owais]

@vytas7 The dependency is very small both in terms of size and functionality but I see your concern. It can be integrated into the framework in a way that does not require the dependency to run but works automatically when it is available. We'd essentially be creating a tiny wrapper for OpenTelemetry in Falcon that uses the real module when available and is noop when not although technically that is already what opentelemetry-api does for the actual library (opentelemetry-sdk).

@owais Do you know any Python frameworks that could serve as an example how to properly ship and integrate it?

Google PubSub Python client is in the process of adding this. PR: googleapis/python-pubsub#149

Happy to help whichever direction you'd like to take.

[vytas7]

That sounds good, yeah, that's what we are doing for certain optional media de/serialization features such as msgpack -- support is optional (but it is exercised in the test suite), and you need to install the relevant dependencies in order to leverage it.

We'll need to take a closer look at all this, but I do like the idea of adding some optional instrumentation affordances directly to the framework, and OpenTelemetry could be the first such plugin. This could hopefully allow us to reduce the instrumentation overhead even further, and make sure Falcon is well supported. See also: #10 & #1908.

[adriangb]

I'm super glad to see this collaboration happening! It's great that @vytas7 is thinking about the issue and that @owais is reaching out. This is OSS at it's best 😄

[CaselIT]

@kgriffs @CaselIT thoughts?

I think it would make sense to have, or at least have easy to use hocks that can be leveraged for this

[vytas7]

Hi again @owais & @srikanthccv,
Just to give you an update. I know our velocity has been less than stellar 😬, but we're still planning to ship native OTEL integration as part of Falcon.

[owais]

@aabmass @ocelotl @lzchen FYI ^

[gmittal22]

@vytas7 I am eager to take on this development, are we still looking to work on this task.

[vytas7]

That's awesome @gmittal22 😎 Are you by chance in Prague EuroPython, and would like to Sprint on this on Saturday, or is it unrelated?

[gmittal22]

No, I am based in Asia. Would want to sprint but how does next week sound?