Add bounded release-admission observability metrics

[coderabbitai]

Summary

Add bounded observability metrics for the release-admission gate.

The gate performs GitHub API requests and Git fetch operations. The release workflow blocks publication when this gate fails. Operators need bounded metrics to identify gate failures and operation latency without recording high-cardinality or sensitive values.

Required changes

• Emit a metric for the overall release-admission gate.
• Emit a metric for each fixed release-admission operation.
• Use only bounded labels: canary, operation, outcome, and error_category.
• Record operation duration without labels for revisions, run IDs, paths, URLs, workflow content, or other unbounded identifiers.
• Define a fixed error-category vocabulary.
• Document the metric names, label values, emission points, and retention or export path.
• Add validation that verifies metric emission, fixed label values, error categorization, and the absence of prohibited high-cardinality labels.

Affected areas

• .github/scripts/require-release-admission-canaries.sh
• .github/workflows/release.yml
• Release-admission tests and operational documentation

Acceptance criteria

• The overall gate and every fixed operation produce bounded metrics.
• Metrics use only canary, operation, outcome, and error_category as labels.
• Duration metrics do not include revision, run ID, path, URL, or other unbounded values as labels.
• Failed operations produce a documented fixed error category.
• Tests cover successful and failed gate paths.
• Documentation defines the metric contract and operator use.

Backlinks

• Source pull request: Propose release-hardening and code-health RFCs #556
• Requester: @leynos
• Source review comment: Propose release-hardening and code-health RFCs #556 (comment)

[leynos]

@CodeRabbit plan

[coderabbitai]

Coding Plan

Summary

Instrument the release-admission gate script with three bounded metrics, using a fixed vocabulary of canary, operation, outcome, and error_category, and a fail-closed posture that maps every failure to a documented error_category.

Emit metrics as JSON lines to a file, upload that file as a workflow artefact, and add a job-summary line, mirroring the mutation-testing reporting precedent.

Validate the emitted metrics with an exact_labels-style allowlist and test the script through Pattern A (subprocess plus fakes, plus a Hypothesis boundedness test) and Pattern C (workflow-YAML contract test).

Document the metric contract as a reviewed ADR and operator guidance, following the adr-013 bounded-telemetry conventions.

Design Choices

Design Choice 1: Does `.github/scripts/require-release-admission-canaries.sh` already exist?

Options Considered:

1. Treat the script as introduced by PR #556 and add only the instrumentation. If the script is absent on the target branch, first add minimal, clearly-marked operation boundaries so the metrics have distinct operations to wrap.
2. Implement the full RFC 0005 admission gate (evidence manifest, three-way checksum, scan-freshness) as part of this ticket, then instrument it.

Chosen Option: 1

Rationale: The ticket scope is bounded to observability metrics, so instrument the gate script and add only the operation-boundary scaffolding that the metrics require. Do not build the full admission gate.

The ticket describes the gate as an operating script that performs GitHub API and Git fetch operations, but the working tree contains no such script and no .github/scripts/ directory. The ticket is a follow-up review comment on PR #556, so the script is most likely introduced by that PR branch.

Design Choice 2: Which emission and retention/export path must the shell script use?

Options Considered:

1. Write bounded metric records as JSON lines to a metrics file, upload that file as a workflow artefact, and append a short human-readable summary to $GITHUB_STEP_SUMMARY.
2. Write only to $GITHUB_STEP_SUMMARY.
3. Emit structured lines to stdout only.

Chosen Option: 1

Rationale: This option mirrors the existing mutation-testing reporting precedent (job summary plus downloadable artefact) and produces a durable, parseable record that the allowlist test can validate.

The shell script cannot use the Rust metrics crate. The repository has no metrics backend and no existing GITHUB_STEP_SUMMARY usage.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Instrument the release-admission gate with bounded metrics

This phase adds the bounded metric vocabulary and the emission mechanism to the gate script. It wraps each fixed operation and the overall gate so each produces a bounded metric record. It enforces a fail-closed posture and forbids unbounded values in labels.

Task 1: Define the fixed metric vocabulary

Establish the single source of truth for metric names and bounded label values.

• Add a header block in .github/scripts/require-release-admission-canaries.sh that declares three metric names: netsuke_release_admission_gate_total (counter), netsuke_release_admission_operation_total (counter), and netsuke_release_admission_operation_duration_seconds (histogram).
• Declare fixed shell constants for the operation value set (one value per distinct GitHub API request and Git fetch operation the gate performs, for example resolve_tag_commit, fetch_workflow_run, check_scan_freshness, verify_evidence).
• Declare the fixed outcome value set as success, failure, and unknown.
• Declare the fixed canary value set as a bounded enumeration that names the canary/freshness check (for example history_scan or none), not a free-form identifier.
• Declare the fixed error_category value set (for example none, api_error, fetch_error, stale_evidence, missing_evidence, mismatch, timeout, unknown). Use none on success and unknown on any unclassified failure.
• Follow the naming convention from docs/adr-013: netsuke_ prefix, snake_case, _total for counters, _duration_seconds for histograms.

Task 2: Add a bounded metric-emission helper

Add one helper function that writes structured metric records and refuses unbounded input.

• Add an emit_metric shell function that accepts a metric name, a bounded label set, and a value, then appends one JSON line to a metrics file path resolved from an environment variable (default to a path under the runner temp directory).
• Validate every label value against the declared value sets before writing. On an out-of-vocabulary value, fail closed and record outcome=unknown, error_category=unknown rather than writing the raw value.
• Restrict the duration record netsuke_release_admission_operation_duration_seconds to the bounded operation label only. Never place a revision, run ID, path, URL, or workflow content in any label.
• Keep the helper aligned with the existing script conventions: set -euo pipefail, printed diagnostics on failure, and outputs surfaced through $GITHUB_OUTPUT.

Task 3: Wrap each fixed operation and the overall gate

Instrument the operation boundaries and the gate result.

• Wrap each fixed operation (each GitHub API request and each Git fetch) with a monotonic start/stop timing measurement and a classification step that maps the result to a bounded outcome and error_category.
• After each operation, emit netsuke_release_admission_operation_total with canary, operation, outcome, and error_category, and emit netsuke_release_admission_operation_duration_seconds with the bounded operation label and the measured seconds.
• At the end of the gate, emit netsuke_release_admission_gate_total with the overall outcome and error_category.
• Preserve the fail-closed posture: a missing, stale, or unclassified result maps to outcome=failure or outcome=unknown and a documented error_category, and the gate still exits non-zero.
• If the target branch has no admission script, add the minimal operation boundaries this instrumentation wraps, and mark them clearly as scaffolding for the RFC 0005 admission gate.

🤖 Prompt for AI agents

Implement bounded observability instrumentation in the release-admission gate
script `.github/scripts/require-release-admission-canaries.sh`. The goal is a
fixed metric vocabulary, a safe emission helper, and full operation-boundary
wrapping.

First, add a header block declaring three metric names:
`netsuke_release_admission_gate_total` (counter),
`netsuke_release_admission_operation_total` (counter), and
`netsuke_release_admission_operation_duration_seconds` (histogram). Follow the
`docs/adr-013` naming convention: `netsuke_` prefix, `snake_case`, `_total` for
counters, `_duration_seconds` for histograms.

Declare fixed shell constants for each label value set:
- `operation`: one value per distinct GitHub API request and Git fetch operation
(for example `resolve_tag_commit`, `fetch_workflow_run`, `check_scan_freshness`,
`verify_evidence`).
- `outcome`: `success`, `failure`, `unknown`.
- `canary`: a bounded enumeration naming the canary/freshness check (for example
`history_scan` or `none`), never a free-form identifier.
- `error_category`: a fixed set (for example `none`, `api_error`, `fetch_error`,
`stale_evidence`, `missing_evidence`, `mismatch`, `timeout`, `unknown`); use
`none` on success and `unknown` on any unclassified failure.

Add an `emit_metric` shell function that accepts a metric name, a bounded label
set, and a value, and appends one JSON line to a metrics file path resolved from
an environment variable (default to a path under the runner temp directory).
Validate every label value against the declared vocabulary before writing; on an
out-of-vocabulary value, fail closed and record `outcome=unknown`,
`error_category=unknown` instead of writing the raw value. Restrict the duration
metric to the bounded `operation` label only, and never place a revision, run
ID, path, URL, or workflow content in any label. Keep the helper aligned with
existing script conventions: `set -euo pipefail`, printed diagnostics on
failure, and outputs surfaced through `$GITHUB_OUTPUT`.

Wrap each fixed operation (each GitHub API request and each Git fetch) with a
monotonic start/stop timing measurement and a classification step mapping the
result to a bounded `outcome` and `error_category`. After each operation, emit
`netsuke_release_admission_operation_total` with `canary`, `operation`,
`outcome`, `error_category`, and emit
`netsuke_release_admission_operation_duration_seconds` with the bounded
`operation` label and the measured seconds. At the end of the gate, emit
`netsuke_release_admission_gate_total` with the overall `outcome` and
`error_category`. Keep the fail-closed posture: a missing, stale, or
unclassified result maps to `outcome=failure` or `outcome=unknown` with a
documented `error_category`, and the gate still exits non-zero.

If `.github/scripts/require-release-admission-canaries.sh` does not exist on the
target branch, add the minimal operation boundaries this instrumentation wraps,
and mark them clearly as scaffolding for the RFC 0005 admission gate. Do not
build the full admission gate.

Phase 2: Wire metric emission into the release workflow

This phase connects the instrumented script to the release workflow. It exposes the metrics file and preserves least-privilege permissions.

Surface and upload the metrics artefact

Make the emitted metrics available to operators and to the tests.

• Update .github/workflows/release.yml so the admission step sets the metrics file path environment variable and runs the instrumented script.
• Add a step that uploads the metrics file as a workflow artefact, mirroring the mutation-testing reporting precedent (job summary plus downloadable artefact).
• Append a short human-readable summary of the gate outcome to $GITHUB_STEP_SUMMARY. Document this as a new, deliberate use of the job summary, because no such usage exists in the repository today.
• Preserve the existing least-privilege permissions blocks and the existing action SHA pins. Do not widen any permission.

🤖 Prompt for AI agents

Wire the instrumented gate script into `.github/workflows/release.yml` so the
metrics it emits reach operators and tests.

Update the admission step to set the metrics file path environment variable and
to run the instrumented script. Add a step that uploads the metrics file as a
workflow artefact, mirroring the mutation-testing reporting precedent of a job
summary plus a downloadable artefact.

Append a short human-readable summary of the gate outcome to
`$GITHUB_STEP_SUMMARY`. Treat this as a new, deliberate use of the job summary,
since no repository workflow uses it today.

Preserve the existing least-privilege `permissions` blocks and the existing
action SHA pins exactly as they are. Do not widen any permission.

Phase 3: Validation and tests

This phase adds the allowlist validator and the tests. It verifies emission, fixed label values, error categorisation, and the absence of prohibited high-cardinality labels.

Task 1: Add a metric allowlist validator

Provide the single audit point for the shell-emitted metric vocabulary.

• Add a Python allowlist module under tests/workflow_contracts/ that mirrors the exact_labels/ConfigMetricsRecorder shape from src/observability_recorder.rs.
• Declare the fixed metric-name set, and for each name an ordered (label, allowed_values) mapping matching the vocabulary from Task 1.1.
• Implement an exact_labels-style check that parses the emitted JSON lines and rejects unknown metric names, extra or missing labels, wrong label order, and out-of-vocabulary values.

Task 2: Add subprocess behaviour tests (Pattern A)

Exercise the real script through fakes and assert on emitted metrics.

• Add a pytest module under scripts/tests/ that follows test_check_markdown_format.py.
• Write fake gh and git executables to a temp directory, add them to PATH, and record their invocations in a JSON-lines call log.
• Run the real script with subprocess.run(..., capture_output=True, check=False, env=...). Cover the success path and the failure paths (API error, fetch error, stale evidence, missing evidence).
• Assert on returncode, on the emitted metric records through the Task 3.1 validator, and on the fixed outcome and error_category for each path.

Task 3: Add a boundedness property test

Prove the label vocabulary stays bounded for arbitrary inputs.

• Add a Hypothesis property test in the same module, using @given and @settings(deadline=None), following test_accepts_only_uniform_canonical_line_endings.
• Generate arbitrary revisions, run IDs, paths, and URLs as fake inputs. Assert that every emitted metric label value is drawn only from the fixed vocabulary and that no generated identifier appears in any label.

Task 4: Pin the workflow wiring (Pattern C)

Pin how the release workflow invokes the instrumented script.

• Add a workflow-contract test under tests/workflow_contracts/ that uses workflow_loading.py helpers (load_workflow, job_steps, step_index_by_key).
• Assert the admission step's permissions, its position relative to the publishing step, its metrics-path argument, and the presence of the artefact-upload step.

🤖 Prompt for AI agents

Build the test suite that verifies the bounded-metric instrumentation from
Phases 1 and 2.

First, add a Python allowlist module under `tests/workflow_contracts/` that
mirrors the `exact_labels`/`ConfigMetricsRecorder` shape from
`src/observability_recorder.rs`. Declare the fixed metric-name set, and for each
name an ordered `(label, allowed_values)` mapping matching the vocabulary
defined in Phase 1. Implement an `exact_labels`-style check that parses the
emitted JSON lines and rejects unknown metric names, extra or missing labels,
wrong label order, and out-of-vocabulary values.

Second, add a pytest module under `scripts/tests/` that follows the structure of
`test_check_markdown_format.py`. Write fake `gh` and `git` executables to a temp
directory, add them to `PATH`, and record their invocations in a JSON-lines call
log. Run the real script with `subprocess.run(..., capture_output=True,
check=False, env=...)`, covering the success path and the failure paths (API
error, fetch error, stale evidence, missing evidence). Assert on `returncode`,
on the emitted metric records through the allowlist validator, and on the fixed
`outcome` and `error_category` for each path.

Third, in the same module, add a Hypothesis property test using `@given` and
`@settings(deadline=None)`, following the pattern of
`test_accepts_only_uniform_canonical_line_endings`. Generate arbitrary
revisions, run IDs, paths, and URLs as fake inputs, and assert that every
emitted metric label value is drawn only from the fixed vocabulary and that no
generated identifier appears in any label.

Fourth, add a workflow-contract test under `tests/workflow_contracts/` using the
`workflow_loading.py` helpers `load_workflow`, `job_steps`, and
`step_index_by_key`. Assert the admission step's permissions, its position
relative to the publishing step, its metrics-path argument, and the presence of
the artefact-upload step.

Phase 4: Documentation

This phase documents the metric contract, the emission points, and the retention/export path, and records the design as a reviewed contract.

Document the metric contract and operator use

Define the stable contract and how operators use it.

• Add an ADR (for example docs/adr-014-release-admission-observability.md) that follows the structure of adr-013: Status, Date, Context, Decision, Rationale, Consequences, Alternatives, and Implementation references.
• Document each metric with the bullet-list "instrument" convention from docs/developers-guide.md: name in backticks, kind, purpose, labels with bounded values (key=value1|value2), and operator use.
• State the fixed error_category vocabulary, the fail-closed mapping, and the rule that renaming a metric or label is a breaking change.
• Document the retention/export path: JSON-lines metrics file uploaded as a workflow artefact plus a job-summary line, and state explicitly that this is not a Prometheus, OTLP, or statsd endpoint.
• Add an operator-facing section to docs/developers-guide.md (or the release operator guidance) that explains how to read the gate and operation metrics.
• Update the roadmap references in docs/roadmap.md §9.4 to link the observability work.

🤖 Prompt for AI agents

Write the reviewed documentation for the bounded-metric contract established in
Phases 1–3.

Add an ADR, for example `docs/adr-014-release-admission-observability.md`,
following the structure of `adr-013`: Status, Date, Context, Decision,
Rationale, Consequences, Alternatives, and Implementation references.

Document each metric using the bullet-list "instrument" convention from
`docs/developers-guide.md`: name in backticks, kind, purpose, labels with
bounded values written as `key=value1|value2`, and operator use. State the fixed
`error_category` vocabulary, the fail-closed mapping, and the rule that renaming
a metric or label is a breaking change.

Document the retention/export path: a JSON-lines metrics file uploaded as a
workflow artefact, plus a job-summary line. State explicitly that this is not a
Prometheus, OTLP, or statsd endpoint.

Add an operator-facing section to `docs/developers-guide.md` (or equivalent
release operator guidance) that explains how to read the gate and operation
metrics.

Update the roadmap references in `docs/roadmap.md` §9.4 to link the
observability work.

Research

Netsuke is a Rust build-system compiler with an extensive CI layer under .github/workflows/. The release path uses release.yml (metadata → build matrix → smoke test → gated release job) and release-dry-run.yml. The release-admission gate is described only in the proposed RFC docs/rfcs/0005-release-hardening.md and in unchecked roadmap tasks (§9.4); the working tree contains no .github/scripts/ directory and no admission or canary script.

The repository holds a mature bounded-cardinality telemetry pattern in src/observability.rs and src/observability_recorder.rs (fixed metric-name constants, small enums for label values, an exact_labels allowlist, local-recorder tests), governed by adr-009 and adr-013.

It uses two established shell-script test patterns: Pattern A (pytest + subprocess + fake executable with a JSON-lines call log, plus a Hypothesis property test) in scripts/tests/test_check_markdown_format.py, and Pattern C (workflow-YAML contract tests) in tests/workflow_contracts/ via workflow_loading.py.

The repository has no Prometheus, OTLP, statsd, or Datadog integration; the mutation-testing workflow reports through the job summary and downloadable artefacts.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Implement bounded observability instrumentation in the release-admission gate
script `.github/scripts/require-release-admission-canaries.sh`. The goal is a
fixed metric vocabulary, a safe emission helper, and full operation-boundary
wrapping.

First, add a header block declaring three metric names:
`netsuke_release_admission_gate_total` (counter),
`netsuke_release_admission_operation_total` (counter), and
`netsuke_release_admission_operation_duration_seconds` (histogram). Follow the
`docs/adr-013` naming convention: `netsuke_` prefix, `snake_case`, `_total` for
counters, `_duration_seconds` for histograms.

Declare fixed shell constants for each label value set:
- `operation`: one value per distinct GitHub API request and Git fetch operation
(for example `resolve_tag_commit`, `fetch_workflow_run`, `check_scan_freshness`,
`verify_evidence`).
- `outcome`: `success`, `failure`, `unknown`.
- `canary`: a bounded enumeration naming the canary/freshness check (for example
`history_scan` or `none`), never a free-form identifier.
- `error_category`: a fixed set (for example `none`, `api_error`, `fetch_error`,
`stale_evidence`, `missing_evidence`, `mismatch`, `timeout`, `unknown`); use
`none` on success and `unknown` on any unclassified failure.

Add an `emit_metric` shell function that accepts a metric name, a bounded label
set, and a value, and appends one JSON line to a metrics file path resolved from
an environment variable (default to a path under the runner temp directory).
Validate every label value against the declared vocabulary before writing; on an
out-of-vocabulary value, fail closed and record `outcome=unknown`,
`error_category=unknown` instead of writing the raw value. Restrict the duration
metric to the bounded `operation` label only, and never place a revision, run
ID, path, URL, or workflow content in any label. Keep the helper aligned with
existing script conventions: `set -euo pipefail`, printed diagnostics on
failure, and outputs surfaced through `$GITHUB_OUTPUT`.

Wrap each fixed operation (each GitHub API request and each Git fetch) with a
monotonic start/stop timing measurement and a classification step mapping the
result to a bounded `outcome` and `error_category`. After each operation, emit
`netsuke_release_admission_operation_total` with `canary`, `operation`,
`outcome`, `error_category`, and emit
`netsuke_release_admission_operation_duration_seconds` with the bounded
`operation` label and the measured seconds. At the end of the gate, emit
`netsuke_release_admission_gate_total` with the overall `outcome` and
`error_category`. Keep the fail-closed posture: a missing, stale, or
unclassified result maps to `outcome=failure` or `outcome=unknown` with a
documented `error_category`, and the gate still exits non-zero.

If `.github/scripts/require-release-admission-canaries.sh` does not exist on the
target branch, add the minimal operation boundaries this instrumentation wraps,
and mark them clearly as scaffolding for the RFC 0005 admission gate. Do not
build the full admission gate.
===============================================================================

Task: 2

Wire the instrumented gate script into `.github/workflows/release.yml` so the
metrics it emits reach operators and tests.

Update the admission step to set the metrics file path environment variable and
to run the instrumented script. Add a step that uploads the metrics file as a
workflow artefact, mirroring the mutation-testing reporting precedent of a job
summary plus a downloadable artefact.

Append a short human-readable summary of the gate outcome to
`$GITHUB_STEP_SUMMARY`. Treat this as a new, deliberate use of the job summary,
since no repository workflow uses it today.

Preserve the existing least-privilege `permissions` blocks and the existing
action SHA pins exactly as they are. Do not widen any permission.
===============================================================================

Task: 3

Build the test suite that verifies the bounded-metric instrumentation from
Phases 1 and 2.

First, add a Python allowlist module under `tests/workflow_contracts/` that
mirrors the `exact_labels`/`ConfigMetricsRecorder` shape from
`src/observability_recorder.rs`. Declare the fixed metric-name set, and for each
name an ordered `(label, allowed_values)` mapping matching the vocabulary
defined in Phase 1. Implement an `exact_labels`-style check that parses the
emitted JSON lines and rejects unknown metric names, extra or missing labels,
wrong label order, and out-of-vocabulary values.

Second, add a pytest module under `scripts/tests/` that follows the structure of
`test_check_markdown_format.py`. Write fake `gh` and `git` executables to a temp
directory, add them to `PATH`, and record their invocations in a JSON-lines call
log. Run the real script with `subprocess.run(..., capture_output=True,
check=False, env=...)`, covering the success path and the failure paths (API
error, fetch error, stale evidence, missing evidence). Assert on `returncode`,
on the emitted metric records through the allowlist validator, and on the fixed
`outcome` and `error_category` for each path.

Third, in the same module, add a Hypothesis property test using `@given` and
`@settings(deadline=None)`, following the pattern of
`test_accepts_only_uniform_canonical_line_endings`. Generate arbitrary
revisions, run IDs, paths, and URLs as fake inputs, and assert that every
emitted metric label value is drawn only from the fixed vocabulary and that no
generated identifier appears in any label.

Fourth, add a workflow-contract test under `tests/workflow_contracts/` using the
`workflow_loading.py` helpers `load_workflow`, `job_steps`, and
`step_index_by_key`. Assert the admission step's permissions, its position
relative to the publishing step, its metrics-path argument, and the presence of
the artefact-upload step.
===============================================================================

Task: 4

Write the reviewed documentation for the bounded-metric contract established in
Phases 1–3.

Add an ADR, for example `docs/adr-014-release-admission-observability.md`,
following the structure of `adr-013`: Status, Date, Context, Decision,
Rationale, Consequences, Alternatives, and Implementation references.

Document each metric using the bullet-list "instrument" convention from
`docs/developers-guide.md`: name in backticks, kind, purpose, labels with
bounded values written as `key=value1|value2`, and operator use. State the fixed
`error_category` vocabulary, the fail-closed mapping, and the rule that renaming
a metric or label is a breaking change.

Document the retention/export path: a JSON-lines metrics file uploaded as a
workflow artefact, plus a job-summary line. State explicitly that this is not a
Prometheus, OTLP, or statsd endpoint.

Add an operator-facing section to `docs/developers-guide.md` (or equivalent
release operator guidance) that explains how to read the gate and operation
metrics.

Update the roadmap references in `docs/roadmap.md` §9.4 to link the
observability work.

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---