-
Notifications
You must be signed in to change notification settings - Fork 991
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Guide for sending Native OpenTelemetry Metrics to Prometheus #2382
base: main
Are you sure you want to change the base?
Add Guide for sending Native OpenTelemetry Metrics to Prometheus #2382
Conversation
Signed-off-by: Prathamesh Sonpatki <csonpatki@gmail.com>
20e4d5f
to
f0ddb2a
Compare
- Configuration changes required in Prometheus setup to enable the ingestion of OpenTelemetry Metrics. | ||
- Setting up a Python application with OpenTelemetry SDK. | ||
- Emitting OpenTelemetry metrics via auto-instrumentation. | ||
- Ingesting those metrics in Prometheus natively. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is a lot happening here. It's nice to have a complete example, but for the purpose of demonstrating Prometheus, isn't this a bit too much that is unrelated to the topic at hand?
Setting up flask app and auto-instrumentations feels like out of scope for basic tutorial here? I believe a basic cli that pushes manually collected metrics once should be a more appropriate level. Then linking to opentelemetry tutorials for more advanced examples.
Especially considering this could be read by someone who doesn't come from Python background.
What do others think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gouthamve Thoughts?
Let's add the OpenTelemetry instrumentation now. We will use Otel's Flask instrumentation library which auto-instruments all request handelers. | ||
|
||
``` shell | ||
pip opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-instrumentation-flask |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be nice if example included creation of virtualenv. Installing to system python is against best practice. In newer versions of ubuntu, pip install fails and asks to provide --break-system-packages
arg.
- We setup required `reader`, `resource` and `provider` for metrics instrumentation. | ||
- We wrapped the Flask app in `FlaskInstrumentor's` `instrument_app` method so that the auto instrumentation will kick in. | ||
|
||
When the application is run and few requests are sent to it, the auto instrumented metrics will be **pushed** from the application to Prometheus via the `FlaskInstrumentor`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This can be misunderstood as its the FlaskInstrumentor that pushes the metrics. If i understand correctly, what happens is that FlaskInstrumentor only collects the metrics internally. PeriodicExportingMetricReader+OTLPMetricExporter does the push.
|
||
## Custom Instrumentation | ||
|
||
Let's add a custom metric in the Python application to count number of calculaor operations performed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
typo calculator
No description provided.