Export the Neutron error-kind breakdown as a dedicated OTLP counter#29
Merged
Merged
Conversation
Add a sixth OTLP instrument, the monotonic counter
openstack_tester.operation.errors with attributes kind, operation, and
error.kind, recorded from Telemetry.RecordOperation whenever errKind is
non-empty. The seam already receives the neutron client's verbatim
errKind classification, so no call sites in internal/neutron change and
the five existing instrument families keep their exact schema.
operation.duration deliberately collapses failures to
outcome ∈ {success, error, timeout}; the counter preserves the
per-error-kind breakdown a metrics backend needs to tell a quota
rejection from a 503 from a 409 conflict, matching the report's Errors
table. A counter adds one cheap series per (kind, operation, error.kind)
that actually occurs, and only on failures, rather than multiplying the
histogram's ~15 streams by the error-kind set.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Christian Berendt <berendt@b42labs.com>
Add the openstack_tester.operation.errors row to README §9's instrument table, document error.kind's bounded value set (quota, timeout, canceled, other, http_<status>) and its Prometheus surface (openstack_tester_operation_errors_total with label error_kind), note it is recorded only for failed operations, and add the by-error-kind cookbook query. Record on the otel-verify target — in the Makefile comment and README — that this family is deliberately absent from the required set: a healthy run produces no errors, so its absence is the expected steady state, not a missing-metric failure. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Christian Berendt <berendt@b42labs.com>
The #24 dashboards landed first, so #25 picks up the breakdown panel the error panels were forward-pointing at. Add a full-width "Errors by error kind" stacked timeseries to the Neutron API operations dashboard, driven by openstack_tester_operation_errors_total summed by kind, operation, and error_kind — the report's Errors table over time, empty on a healthy run. Retire the "error kinds collapse to outcome by design" / "lands with the dedicated errors counter" forward-pointers in the api-operations and overview dashboard descriptions and in README §9, now that the counter and its panel exist: the outcome panels still split by outcome, the new panel breaks failures down by error_kind. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Christian Berendt <berendt@b42labs.com>
berendt
marked this pull request as ready for review
July 2, 2026 18:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #25
Summary
The OTLP export from #19 collapses every failure to
outcome ∈ {success, error, timeout}onopenstack_tester.operation.duration, so a metrics backend can see that Neutron calls fail but not why. This branch exports the neutron client'serrKindclassification as a dedicated counter,openstack_tester.operation.errors, so a PromQL backend can answer "which kinds of errors, on which resource kind and operation, at what rate" — the run report's Errors table, over time. The five existing instrument families keep their exact schema; the counter is a sixth instrument that fires only on failures.Change set (in commit order)
a963904— Export failed Neutron operations as an errors-by-kind counterinternal/telemetry/telemetry.go: the monotonic counteropenstack_tester.operation.errorswith attributeskind,operation, anderror.kind(Prometheus:openstack_tester_operation_errors_total, labelerror_kind).Telemetry.RecordOperationwhenevererrKind != "". The seam already receives the neutron client's verbatimerrKind, so no call sites ininternal/neutronchange and the existing five instrument families are byte-identical in schema.error.kindattribute on the histogram: it adds one cheap series per(kind, operation, error_kind)that actually occurs, only on failures, instead of multiplying the histogram's ~15-series streams by the error-kind set — and it leaves the documentedoperation.durationschema untouched.telemetry_test.goasserts the emitted schema via the ManualReader provider (a failing op yields the counter with the expected attributes; a success yields nothing);neutron/telemetry_test.gocovers the classification end-to-end including aquota, anhttp_<status>, and a success case.8be04c7— Document the operation errors counter in README and otel-verifyopenstack_tester.operation.errorsrow to README §9's instrument table, documentserror.kind's bounded value set (quota,timeout,canceled,other,http_<status>) and its Prometheus surface, notes it is recorded only for failed operations, and adds the by-error-kind cookbook query.Makefilecomment and README — that this family is deliberately absent from themake otel-verifyrequired set: a healthy run produces no errors, so its absence is the expected steady state, not a missing-metric failure.65e6e29— Add an errors-by-kind panel to the API operations dashboardcontrib/otel/dashboards/api-operations.json), driven byopenstack_tester_operation_errors_totalsummed bykind,operation, anderror_kind— the report's Errors table over time, empty on a healthy run.api-operationsandoverviewdashboard descriptions and in README §9, now that the counter and its panel exist.Divergence from the issue
None material. The issue anticipated (step 5) that "whichever issue lands second picks up" the dashboard breakdown; #24's dashboards landed first, so
65e6e29supplies the panel here. All four acceptance criteria are covered: the new counter fires with the right attributes on failure and not on success, the existing five families are schema-identical, README §9 documents the instrument and cookbook query, and unit tests coverquota,http_<status>, and success cases.Implemented by planwerk-agent 6b3f3ea with Claude:claude-opus-4-8