Skip to content

Every validator is pinned to stacks/observability — a second stack would be checked by nothing #263

Description

@Gerrrt

Split out of #101, which
recorded that "the Makefile is already stack-parameterised (STACK ?= observability), so the tooling supports it."

Half of that is true, and it is the half that does not matter. STACK ?=
reaches the lifecycle and secrets targets — up, down, render,
secrets-edit, reload. Every validator is pinned to the one directory:

Pinned to stacks/observability
scripts/validate.sh:17 STACK="stacks/observability"
scripts/check_docs.py:72 STACK = REPO / "stacks/observability"
scripts/check_dashboards.py:37,38,56 dashboards, datasources, alloy dir
scripts/check_compose_health.py:63 DEFAULT = ...
scripts/check_loki_rules.sh:28 STACK=
scripts/seed-validation-env.sh:26 STACK=
.github/workflows/ci.yml:18 STACK: stacks/observability

make check-rules is the exception that proves it: it uses $(STACK_DIR) and
so would follow STACK=lab, while make validate — which is what CI runs and
what a contributor runs — would not.

A stacks/lab/ created before this work is a stack CI has never seen. Its
compose is not config-checked, its rules are not promtool-tested, its
dashboards are not checked against its own datasources, its healthcheck
binaries are not probed, and its images are pinned and digest-verified by
nothing. Every guard this repository has built for the estate's stack would be
silently absent for the lab's — which is the shape of #62, #65, #68 and #79 all
over again, in a directory nobody is watching.

Worth deciding

  • Iterate, or matrix? A for stack in stacks/*/ inside validate.sh is
    one change and keeps a single CI job. A GitHub Actions matrix gives a red
    square per stack and a clearer failure, at the cost of the job list growing
    with the estate.
  • What is genuinely per-stack and what is repo-wide. check_docs.py
    asserts prose against configs and much of it is estate-specific — the SNMP
    targets, the ports table, the compute table. Those are not "run it twice"
    candidates; splitting repo-wide assertions from per-stack ones is most of the
    work here.
  • What a stack is allowed to lack. The lab has no snmp-exporter,
    no blackbox-exporter and no Alertmanager (ADR-0019), so every checker needs
    to distinguish "absent by design" from "missing". render-config.sh already
    guards each render behind an if [[ -f ... ]]; its REQUIRED array does
    not, and today demands all four SNMP communities and all four Alertmanager
    URLs from any stack.

Done when

make validate and CI both check every directory under stacks/, and adding
an empty one fails loudly rather than passing quietly.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions