alertlint

A skill/CLI that analyzes a service's alert configurations alongside its prior alerting history — and the actions taken (or not taken) for each alert — to score services and recommend concrete improvements.

What it does

alertlint looks at three inputs for a service:

1. Alert configurations — the alerts currently defined for the service (thresholds, conditions, severities, routing).
2. Alerting history — when alerts fired, how often, for how long, and at what times.
3. Action taken — what responders actually did when each alert fired (acknowledged, remediated, silenced, ignored).

From these it produces:

• A service score — noise, coverage, and threshold-quality sub-scores rolled into a criticality-weighted priority score, identical whether a service is scored alone or among thousands (no corpus statistics ever enter).
• A prioritized worklist — the actual deliverable: where to spend engineering attention first.
• Typed, evidence-backed findings — noise | coverage | threshold | identity, greppable with jq, each carrying the raw facts that produced it.
• Concrete proposed changes — threshold updates, routing changes, missing-alert additions; generated by the bundled Claude skill, never applied automatically.

Install

Download a static binary from GitHub Releases (macOS/Linux/Windows, amd64/arm64, checksums attached), or build from source:

go install github.com/davetashner/alertlint/cmd/alertlint@latest

Quickstart (no credentials needed)

The repo ships a replayable demo corpus:

# Score two demo services offline
go run ./cmd/alertlint analyze \
  --replay fixtures/demo --tenant demo --out out \
  --identity-conventions fixtures/demo/identity-conventions.yaml

# Rank them
go run ./cmd/alertlint worklist out
# rank  ci_id      ci_name       tier  priority  composite  findings
# 1     CI0002222  payments-api  2     76.9      48.7       4
# 2     CI0001111  checkout-api  1     67.0      66.5       3

# Pin a fuzzy identity suggestion so the next run joins it (the ratchet)
go run ./cmd/alertlint identity confirm newrelic/policy/998811 CI0002222 \
  --mappings identity-mappings.yaml --by you@example.com

docs/dryrun/mvp-dry-run.md walks the full acceptance scenario, including the skill adjudicating a self-healing alert.

Running against real sources

analyze registers adapters from environment credentials — you analyze whatever your own credentials can reach (there is no central broker):

Source	Provides	Credentials
Datadog	alert configs	DD_API_KEY, DD_APP_KEY
New Relic	alert configs	NEW_RELIC_API_KEY
CloudWatch	alert configs	AWS default chain (AWS_REGION / AWS_PROFILE)
Splunk	alert configs	SPLUNK_URL, SPLUNK_TOKEN
PagerDuty	firing history + responses	PAGERDUTY_TOKEN
ServiceNow	history + responses + CMDB CIs	SERVICENOW_URL, SERVICENOW_USER, SERVICENOW_PASSWORD

alertlint analyze --tenant my-org --out out \
  --identity-conventions my-conventions.yaml \
  --identity-mappings identity-mappings.yaml \
  --archetype-overrides archetype-overrides.yaml   # optional, skill/human assertions
alertlint worklist out

Convention rules (--identity-conventions) are per-estate config: see configs/identity-conventions.yaml for the format and fixtures/demo/identity-conventions.yaml for a worked set.

How it works

adapters (Datadog, NR, CW, Splunk, PD, SNOW)
   |  canonical records + verbatim identity hints
   v
identity resolver (exact -> confirmed -> convention; fuzzy = suggestions only)
   |  artifact->CI join table + coverage metrics
   v
scoring engine (deterministic: disposition table, cold-start gating,
   |            noise/coverage/threshold sub-scores, priority formula)
   v
per-service JSON documents (the stable contract) --> worklist aggregator
   |
   v
Claude skill (skills/alertlint): triages low-confidence findings,
enriches archetypes from repos, proposes concrete changes -- never
recomputes, never adjusts scores

Design principles (see docs/decision-records):

• The CLI never calls an LLM; the skill never recomputes statistics (ADR 0003). Low-confidence findings are the designed handoff between them.
• Per-service scoring is the atomic unit (ADR 0001) — the worklist merges corpora from any number of callers without rescoring.
• Fuzzy identity matches never join scoring (ADR 0002) — they surface as candidates until explicitly confirmed.
• Everything is reproducible (Go, byte-identical output, versioned configs, snapshot replay — ADR 0005).

Repository layout

cmd/alertlint/          # CLI: analyze, worklist, identity confirm
internal/               # adapters, identity, archetype, score, output, pipeline
archetypes/             # the versioned archetype -> required-signal library
configs/                # scoring config, starter identity conventions
skills/alertlint/       # the Claude skill (SKILL.md + references)
fixtures/demo/          # replayable demo corpus
docs/
├── requirements/       # what alertlint must do (REQ-<CATEGORY>-NNN, source of truth)
├── specs/              # how each requirement is implemented
├── decision-records/   # why we chose what we chose (ADRs)
└── dryrun/             # MVP acceptance record

Traceability flows requirement → spec → decision records → implementation; each spec header lists the REQ-<CATEGORY>-NNN IDs it addresses.

Development

gofmt -l . && go vet ./... && go build ./... && go test -race -count=1 ./...
python3 scripts/check_traceability.py       # spec/ADR structure + REQ-ID existence
scripts/check_contract.sh <corpus-dir>      # the output contract's jq acceptance checks

CI runs all of the above plus offline doc-link checking and archetype-library schema validation; the three job names are required checks on main. The backlog is managed with beads in .beads/ — run bd list --status open to see what's next.

License

See LICENSE.