Repository navigation
Roadmap question: OpenTelemetry metrics export for ZenML server #4794
Replies: 7 comments 4 replies
|
Hey @sarangtagad-git, Thanks so much for your message, it is much appreciated 💜 You are right. ZenML ships with an As for the topic of metrics, I think I can summarize the situation under two different features:
For both features, a community contribution is very welcome where the backend changes would be available to OSS users. I would have to discuss the UI/UX implications with the team when it comes to the ZenML Pro. Let me know if this clears your questions and let us know if we can help in any way. |
|
Hi @bcdurak, After studying the This is about feature number 1, that allows people to capture metrics during step/pipeline execution: Proposed:
|
|
Hey @sarangtagad-git, Thank you for this detailed proposal. Here are a few points with some feedback:
As for your questions:
I hope this message clarifies some of the topics. Perhaps you can use this PR (that introduced the Log Store abstraction) as a guideline. Let us know if we can help in any other way. |
|
Hi @bcdurak I have drafted a PR for this feature_1 ( A feature that allows people to capture metrics during step/pipeline execution: ) -> #4854 Can you please review this and provide your early feedback. Thank you in advance 😊 |
|
Hi @bcdurak, I am continuing on feature 2 ( Aggregated Overview) mentioned above by you. Hence, I opened #5274 with a repro and a proposed filter, and I'm happy to implement it. |
|
Hi @sarangtagad-git, I've been following your progress on the OTel metrics implementation and the aggregated overview. Regarding the issue you mentioned in #5274 about deployment invocations skewing the aggregates: A clean way to handle this would be to introduce a run_type or invocation_source attribute to the run metadata. By tagging runs as either pipeline_execution or deployment_invocation, you could implement a simple filter in the statistics endpoint. Specifically, in the aggregation logic, you could add an optional query parameter like ?exclude_deployments=true that filters the underlying database query to only include runs where the source is not a deployment. This would keep the API clean while providing the operational visibility you're looking for. |
|
Thanks @ZitouniNidhal. Just to clarify the approach in #5274: One more limitation of the metadata approach:
I'm starting the implementation now (bool filter on PipelineRunFilter, details in #5274 ) |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hi ZenML team
First off, thank you for all the work on ZenML. I've been digging into it over the last few days and the codebase has been a pleasure to explore. I'm a platform engineer exploring ZenML, and I'm thinking ahead about how we'd integrate it with our existing observability stack (Grafana + Prometheus + AlertManager). Before I sketch anything out, I wanted to check in with you all to make sure I'm not missing existing work or going against the grain of where you're heading.
A few things I noticed while looking around (please correct me if any of this is out of date):
OtelLogStore(and aDatadogLogStore) for exporting step logs via OTLP. Must be a long term planned strategy on observability./metricsendpoint, no OTel metrics signal, nothing turning up underprometheus/otel metricsin the codebase. Please let me know if I missed it.aggregate/operational — trends over time, alerting, SLOs, correlating ZenML activity with infra metrics in our existing dashboards.
I have following idea I wanted to run by you
An
OtelMetricsExporter(stack component or server-side emitter) that mirrors theOtelLogStorepattern and exports metrics via OTLP. Some possible candidate metrics that came to mind Or there could be better options also here.zenml_pipeline_runs_total{status, pipeline}— counterzenml_pipeline_run_duration_seconds{pipeline}— histogramzenml_step_runs_total{status, step, pipeline}— counterzenml_step_duration_seconds{step, pipeline}— histogramzenml_step_cache_hits_total{step, pipeline}— counter (cache visibility seems valuable for ML workloads)zenml_active_runs— gaugeHence, I have following questions for you :
Can you please tell me if this OTel metrics is already on your roadmap?
Would a community contribution be welcome here? And if so, does the stack-component approach (mirroring
OtelLogStore) feel right to you, or would you prefer something different ?How does this fit with ZenML Pro? I'd guess that polished aggregate UI analytics is a Pro concern, and an OTel exporter would mainly help self-hosted OSS users plug ZenML into their existing observability tools. I want to make sure I'd be complementing your direction, not competing with it.
Where would such a contribution live in the repo? A new
metrics_exporters/sibling tolog_stores/, something insidezen_server/, or a layout you'd prefer?Can you please let me know your thoughts on this. Thank you in advance !!!
All reactions