feat(otel): onboard atecontroller to the OTLP path - #754
Merged
Eitan Yarmush (EItanya) merged 2 commits intoAug 6, 2026
Merged
Conversation
Krisztian F (krisztianfekete)
marked this pull request as ready for review
August 5, 2026 14:35
Collaborator
|
Please rebase |
Krisztian F (krisztianfekete)
force-pushed
the
chore/onboard-atecontroller-to-otel
branch
from
August 6, 2026 07:28
2bbb200 to
a531326
Compare
Jeff Luo (JeffLuoo)
approved these changes
Aug 6, 2026
Eitan Yarmush (EItanya)
merged commit Aug 6, 2026
c155efd
into
agent-substrate:main
11 checks passed
Krisztian F (krisztianfekete)
deleted the
chore/onboard-atecontroller-to-otel
branch
August 6, 2026 14:31
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
atecontroller had no OTel at all. Dev-mode zap logger, and its controller-runtime metrics were only available on a :8080 that we don't scrape.
After this PR, logs go through the shared slog handler (plus a
--log-levelflag to match the other binaries), and controller-runtime's Prometheus registry is bridged onto the OTLP reader so the reconcile/workqueue metrics actually reach the collector. Filtering these out is a pipeline responsibility. Also added otelgrpc to the ateapi client, which was untraced.This unblocks #564 the workperpool metrics, cc Angela (@Angelawork), Jeff Luo (@JeffLuoo): there's a working
MeterProviderto use forate.workerpool.desired_workers/ready_workers.Couple of things to mention for review:
-
InitMetricsPushOnly, notInitMetrics, even though we do serve :8080. That port is controller-runtime's own private registry, not the global oneserverboot.metricsMuxserves, so a pull reader there would collect into something we never expose.Bridge is pinned to v0.68.0 to match otelgrpc. Wanted to go to v0.70.0, but that requires otel/sdk/metric 1.45.0 and pulls the whole SDK up with it (406 vendor files instead of 81). Happy to do that bump separately.
zap/zapr fall out of go.mod since atecontroller was the last importer.
I included an OTel collector image bump from the early 2024 (!) one to latest, which was breaking exposing native histograms
Tests pass
Appropriate changes to documentation are included in the PR