Skip to content

Export the Neutron error-kind breakdown as a dedicated OTLP counter#29

Merged
berendt merged 3 commits into
mainfrom
implement/issue-25-operation-errors-counter
Jul 2, 2026
Merged

Export the Neutron error-kind breakdown as a dedicated OTLP counter#29
berendt merged 3 commits into
mainfrom
implement/issue-25-operation-errors-counter

Conversation

@berendt

@berendt berendt commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Closes #25

Summary

The OTLP export from #19 collapses every failure to outcome ∈ {success, error, timeout} on openstack_tester.operation.duration, so a metrics backend can see that Neutron calls fail but not why. This branch exports the neutron client's errKind classification as a dedicated counter, openstack_tester.operation.errors, so a PromQL backend can answer "which kinds of errors, on which resource kind and operation, at what rate" — the run report's Errors table, over time. The five existing instrument families keep their exact schema; the counter is a sixth instrument that fires only on failures.

Change set (in commit order)

a963904 — Export failed Neutron operations as an errors-by-kind counter

  • Adds the sixth OTLP instrument in internal/telemetry/telemetry.go: the monotonic counter openstack_tester.operation.errors with attributes kind, operation, and error.kind (Prometheus: openstack_tester_operation_errors_total, label error_kind).
  • Recorded from Telemetry.RecordOperation whenever errKind != "". The seam already receives the neutron client's verbatim errKind, so no call sites in internal/neutron change and the existing five instrument families are byte-identical in schema.
  • A counter rather than an error.kind attribute on the histogram: it adds one cheap series per (kind, operation, error_kind) that actually occurs, only on failures, instead of multiplying the histogram's ~15-series streams by the error-kind set — and it leaves the documented operation.duration schema untouched.
  • Tests: telemetry_test.go asserts the emitted schema via the ManualReader provider (a failing op yields the counter with the expected attributes; a success yields nothing); neutron/telemetry_test.go covers the classification end-to-end including a quota, an http_<status>, and a success case.

8be04c7 — Document the operation errors counter in README and otel-verify

  • Adds the openstack_tester.operation.errors row to README §9's instrument table, documents error.kind's bounded value set (quota, timeout, canceled, other, http_<status>) and its Prometheus surface, notes it is recorded only for failed operations, and adds the by-error-kind cookbook query.
  • Records — in the Makefile comment and README — that this family is deliberately absent from the make otel-verify required set: a healthy run produces no errors, so its absence is the expected steady state, not a missing-metric failure.

65e6e29 — Add an errors-by-kind panel to the API operations dashboard

  • Adds a full-width "Errors by error kind" stacked timeseries to the Neutron API operations dashboard (contrib/otel/dashboards/api-operations.json), driven by openstack_tester_operation_errors_total summed by kind, operation, and error_kind — the report's Errors table over time, empty on a healthy run.
  • Retires the "error kinds collapse to outcome by design" forward-pointers in the api-operations and overview dashboard descriptions and in README §9, now that the counter and its panel exist.

Divergence from the issue

None material. The issue anticipated (step 5) that "whichever issue lands second picks up" the dashboard breakdown; #24's dashboards landed first, so 65e6e29 supplies the panel here. All four acceptance criteria are covered: the new counter fires with the right attributes on failure and not on success, the existing five families are schema-identical, README §9 documents the instrument and cookbook query, and unit tests cover quota, http_<status>, and success cases.


Implemented by planwerk-agent 6b3f3ea with Claude:claude-opus-4-8

berendt added 3 commits July 2, 2026 19:50
Add a sixth OTLP instrument, the monotonic counter
openstack_tester.operation.errors with attributes kind, operation, and
error.kind, recorded from Telemetry.RecordOperation whenever errKind is
non-empty. The seam already receives the neutron client's verbatim
errKind classification, so no call sites in internal/neutron change and
the five existing instrument families keep their exact schema.

operation.duration deliberately collapses failures to
outcome ∈ {success, error, timeout}; the counter preserves the
per-error-kind breakdown a metrics backend needs to tell a quota
rejection from a 503 from a 409 conflict, matching the report's Errors
table. A counter adds one cheap series per (kind, operation, error.kind)
that actually occurs, and only on failures, rather than multiplying the
histogram's ~15 streams by the error-kind set.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Christian Berendt <berendt@b42labs.com>
Add the openstack_tester.operation.errors row to README §9's instrument
table, document error.kind's bounded value set (quota, timeout, canceled,
other, http_<status>) and its Prometheus surface
(openstack_tester_operation_errors_total with label error_kind), note it
is recorded only for failed operations, and add the by-error-kind
cookbook query.

Record on the otel-verify target — in the Makefile comment and README —
that this family is deliberately absent from the required set: a healthy
run produces no errors, so its absence is the expected steady state, not
a missing-metric failure.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Christian Berendt <berendt@b42labs.com>
The #24 dashboards landed first, so #25 picks up the breakdown panel the
error panels were forward-pointing at. Add a full-width "Errors by error
kind" stacked timeseries to the Neutron API operations dashboard, driven
by openstack_tester_operation_errors_total summed by kind, operation, and
error_kind — the report's Errors table over time, empty on a healthy run.

Retire the "error kinds collapse to outcome by design" / "lands with the
dedicated errors counter" forward-pointers in the api-operations and
overview dashboard descriptions and in README §9, now that the counter
and its panel exist: the outcome panels still split by outcome, the new
panel breaks failures down by error_kind.

Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Christian Berendt <berendt@b42labs.com>
@berendt
berendt marked this pull request as ready for review July 2, 2026 18:01
@berendt
berendt merged commit 4b623c9 into main Jul 2, 2026
2 checks passed
@berendt
berendt deleted the implement/issue-25-operation-errors-counter branch July 2, 2026 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Export the Neutron error-kind breakdown as a dedicated OTLP counter

1 participant