Rebuild: working observability stack, secrets management, CI, and documentation - #1
Merged
Merged
Conversation
The previous layout was <service>/<fqdn>/, five directories sharded by a hostname that had exactly one value and appeared in no documentation. Collapse it to stacks/<stack>/<service>/, where a stack is the set of services deployed together. Also replaces a .gitignore that listed nine hardcoded paths — all of which pointed at files no longer in the tree, one of them misspelled (Mar.pdf for the Mat.pdf that was actually committed) — with pattern-based rules covering OS cruft, secrets, rendered config and runtime state. Rationale in docs/adr/0004-one-compose-stack-per-host.md.
…g stack
None of the previous compose files could start:
* every mount source pointed at a path that does not exist in the repo
(./prometheus.yml vs config/prometheus.yaml, /opt/loki/loki-config.yml,
/opt/alloy/config.alloy)
* grafana's file declared depends_on: [prometheus, loki], neither of which was
defined in it, which compose rejects outright
* grafana mounted its data at /grafana instead of /var/lib/grafana, so its
database was written to the container layer and lost on every recreate
* loki wrote to /tmp/loki while its volume was mounted at /loki, losing every
log line on restart
One compose.yaml now defines all six services on a shared network with
healthchecks and health-gated ordering. Services resolve each other by name,
which removed the hardcoded 10.0.99.20 from both prometheus.yaml and
config.alloy.
Other changes:
* all images pinned to explicit versions; CI fails on :latest
* anonymous Grafana Admin access disabled, password sourced from SOPS
* snmp-exporter is no longer published to a host interface
* Alloy's debug UI binds to 127.0.0.1 only
* Prometheus/Alertmanager/snmp-exporter run as nobody
* Alertmanager added, with severity routing and inhibit rules
* commit generator.yaml, the snmp_exporter source of truth that was missing —
the generated file carried a 'manual changes will be lost' warning with no
way to regenerate it
Fixes a live bug in config.alloy: the log_processor regex used \\b inside a
double-quoted Alloy string, where it is a backspace escape rather than a word
boundary. The regex never matched, so the template's {{ else }}info{{ end }}
fallback labelled every log line 'info'. It is now a backtick string.
The repository committed a plaintext SNMP community string shared across the
firewall, switch, UPS and BMC, plus Grafana admin/admin alongside anonymous
Admin access.
Credentials are now encrypted with age and committed as ciphertext. SOPS leaves
keys in plaintext and encrypts values, so git log shows which credential rotated
and when without revealing what it rotated to. The private key lives at
~/.config/sops/age/keys.txt and never enters the repository.
Each device now has its own SNMP community. SNMPv2c transmits the community in
cleartext, so a single shared string meant one captured packet granted read
access to every device on the network.
snmp_exporter does no environment expansion and reads its config once at
startup, so render-config.sh substitutes the community strings into a gitignored
.rendered/ directory at deploy time. The tracked snmp.yaml keeps its
${PLACEHOLDERS}, and the script fails loudly if any survive substitution.
make secrets-init generates the keypair and creates the encrypted file; it
cannot be pre-seeded here without committing a private key.
Rationale in docs/adr/0005-secrets-with-sops-and-age.md.
Grafana previously had no config at all: no datasources, no dashboards, and
GF_PATHS_PROVISIONING pointing at an empty directory. Nothing about the
visualisation layer was version-controlled.
Adds:
* provisioned Prometheus, Loki and Alertmanager datasources with fixed UIDs
* 5 dashboards / 79 panels — host overview, Docker containers, network and
firewall, UPS and power, logs
* 32 alert rules across host, network, UPS and container concerns
allowUiUpdates is false, so the JSON in git is the source of truth and UI edits
are discarded on restart.
Disk alerting is predictive (predict_linear over 6h, gated on being under 30%
free) rather than a fixed threshold — a disk stable at 86% is not an emergency,
one climbing fast at 60% is.
Also collapses four near-identical SNMP scrape jobs, which differed only in
target/module/auth, into one file_sd job reading targets/snmp.yaml. Targets are
re-read every 5 minutes, so adding a device no longer needs a restart.
The repository is almost entirely YAML and had nothing checking that any of it
was valid. The defects fixed in the preceding commits — an invalid depends_on,
four mount paths pointing at nothing, a regex that silently never matched —
would all have been caught by a pipeline on the day they were introduced.
Three jobs:
lint yamllint, markdownlint, shellcheck, actionlint, editorconfig
validate docker compose config, promtool check config/rules, amtool
check-config, alloy fmt --verify, dashboard structure, and every
dashboard PromQL expression parsed by promtool
secrets gitleaks over the working tree and over full history
Dashboard queries are as easy to typo as alert rules but fail as an empty panel
rather than an error, so all 79 are extracted and parsed.
gitleaks compiles with RE2, so the custom rules avoid lookaheads and exclude
${PLACEHOLDER} values by requiring the value's first character not to be '$'.
The full-history scan honours .gitleaksignore, which enumerates all nine known
historical findings with an explanation of each. That file is an acknowledgement
rather than a fix: a job permanently red for a known reason gets ignored, and
then a genuinely new leak goes unnoticed with it. It gets deleted once the
history purge runs.
Also adds Dependabot for the pinned images, which is what makes pinning
sustainable rather than just stale.
The README was a personal journal entry. It described the lab entirely in the
future tense, listed an objective (red team, Kubernetes, CTF, reverse
engineering) none of which exists in the repository, and contained no
architecture, quick start, or link to anything else. Neither network diagram was
referenced by any file.
Replaces it with a project README, and splits the inventory — which was the best
work in the repository — into documentation that can be navigated:
architecture.md Mermaid topology and data flow that render on GitHub
network.md the per-VLAN inventory, restructured and sanitised
hardware.md rack layout and compute, split out of the inventory
observability.md what is collected, retention, cardinality, operations
security.md threat model, segmentation rationale, secrets handling
roadmap.md the task checkboxes that were buried in the inventory
adr/ 5 decision records
runbooks/ deploy, add a device, rotate credentials, purge history
Sanitisation of network.md: MAC addresses truncated to their OUI, personal
devices listed by role rather than by owner, and camera-to-room mapping removed.
The file previously published the full MAC, OS version and room of every device
in the house, including cameras and a baby monitor, in a public repository. The
vendor half of the MAC keeps everything that was actually useful.
Also drops the $\color{limegreen}{...}$ table headers, which render only on
github.com and make the header text ungreppable.
Mermaid diagrams are now the maintained ones. The existing PNG is 12.5 MB at
9871x4466, which GitHub will not render usefully, and its .drawio source was
deleted in an earlier commit — so it is kept and linked but is no longer the
primary reference.
Adds scripts/purge-history.sh with a --dry-run mode that rewrites a scratch
mirror and reports, leaving the real repository untouched.
There was a problem hiding this comment.
Pull request overview
This PR rebuilds the repo into a deployable “observability stack as code” for the homelab, consolidating previously split compose setups into a single stacks/observability stack with pinned images, provisioned Grafana resources, Prometheus rules, and supporting scripts/docs/CI.
Changes:
- Consolidates observability services into one compose stack with validated configs (Prometheus/Loki/Grafana/Alertmanager/Alloy/snmp-exporter).
- Introduces SOPS+age-based secrets workflow and render/validation tooling (bootstrap, render-config, validate, purge-history).
- Adds CI, extensive documentation (ADRs/runbooks), and cleans up legacy per-service directories.
Reviewed changes
Copilot reviewed 67 out of 68 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| stacks/observability/snmp-exporter/snmp.yaml | Switches SNMP config to placeholder-based generated template. |
| stacks/observability/snmp-exporter/generator.yaml | Adds generator source-of-truth for snmp-exporter config. |
| stacks/observability/README.md | Documents stack layout, edit workflow, and validation commands. |
| stacks/observability/prometheus/targets/snmp.yaml | Adds file_sd SNMP target definitions with labels for alerts. |
| stacks/observability/prometheus/rules/ups.rules.yaml | Adds UPS alert rules. |
| stacks/observability/prometheus/rules/network.rules.yaml | Adds SNMP/network/firewall/switch/iLO alert rules. |
| stacks/observability/prometheus/rules/host.rules.yaml | Adds host-level alert rules (disk/mem/load/clock/reboot). |
| stacks/observability/prometheus/rules/containers.rules.yaml | Adds container + stack self-monitoring rules. |
| stacks/observability/prometheus/prometheus.yaml | New Prometheus config including SNMP relabeling + file_sd. |
| stacks/observability/loki/loki-config.yaml | New Loki single-binary filesystem config with retention. |
| stacks/observability/grafana/provisioning/datasources/datasources.yaml | Provisions Prometheus/Loki/Alertmanager datasources. |
| stacks/observability/grafana/provisioning/dashboards/dashboards.yaml | Provisions dashboards folder/provider settings. |
| stacks/observability/grafana/dashboards/logs-explorer.json | Adds a provisioned Grafana dashboard (Logs). |
| stacks/observability/compose.yaml | New unified compose stack for all observability services. |
| stacks/observability/alloy/config.alloy | Updates Alloy config (regex fix + env-coalesced endpoints). |
| stacks/observability/alertmanager/alertmanager.yaml | Adds Alertmanager routing/inhibition config. |
| stacks/observability/.env.example | Adds non-sensitive tunables for compose interpolation. |
| snmp-exporter/prometheus.matrix.elysium/docker-compose.yaml | Removes legacy per-service compose file. |
| secrets/README.md | Documents secrets workflow and deployment rendering approach. |
| secrets/observability.example.yaml | Adds plaintext template for required secret keys. |
| scripts/validate.sh | Adds local validation script intended to mirror CI checks. |
| scripts/render-config.sh | Adds deploy-time decryption/rendering of env + SNMP config. |
| scripts/purge-history.sh | Adds history-rewrite helper to purge historical secrets. |
| scripts/check_dashboards.py | Adds dashboard JSON/datasource UID validation tool. |
| scripts/bootstrap.sh | Adds bootstrap to generate age keys and create encrypted secrets. |
| README.md | Rewrites repository README with architecture, workflow, and docs links. |
| prometheus/prometheus.matrix.elysium/docker-compose.yaml | Removes legacy Prometheus compose. |
| prometheus/prometheus.matrix.elysium/config/prometheus.yaml | Removes legacy Prometheus config. |
| Makefile | Adds lifecycle, secrets, validation, generation, and backup targets. |
| loki/prometheus.matrix.elysium/docker-compose.yaml | Removes legacy Loki compose. |
| loki/prometheus.matrix.elysium/config/loki-config.yaml | Removes legacy Loki config. |
| LICENSE | Adds MIT license file. |
| grafana/prometheus.matrix.elysium/docker-compose.yaml | Removes legacy Grafana compose. |
| docs/security.md | Adds threat model + secrets/history exposure documentation. |
| docs/runbooks/rotate-snmp-community.md | Adds runbook for rotating SNMP communities. |
| docs/runbooks/purge-git-history.md | Adds runbook for purging secrets from git history. |
| docs/runbooks/deploy-stack.md | Adds deploy runbook for the observability stack. |
| docs/runbooks/add-monitored-device.md | Adds runbook for onboarding Linux hosts/SNMP devices. |
| docs/roadmap.md | Adds consolidated roadmap for follow-up work. |
| docs/observability.md | Documents collection, dashboards, alerting, and operating procedures. |
| docs/network.md | Adds sanitized network inventory and segmentation documentation. |
| docs/images/README.md | Documents screenshot capture/sanitization expectations. |
| docs/hardware/Inventory.md | Removes old inventory doc. |
| docs/hardware.md | Adds rewritten hardware documentation. |
| docs/architecture.md | Adds architecture + dataflow diagrams and port documentation. |
| docs/adr/0005-secrets-with-sops-and-age.md | Adds ADR for secrets approach. |
| docs/adr/0004-one-compose-stack-per-host.md | Adds ADR for stack layout choice. |
| docs/adr/0003-observability-stack-selection.md | Adds ADR for observability tool selection. |
| docs/adr/0002-vlan-segmentation-strategy.md | Adds ADR for trust-based VLAN segmentation. |
| docs/adr/0001-record-architecture-decisions.md | Adds ADR policy for recording decisions. |
| alloy/prometheus.matrix.elysium/docker-compose.yaml | Removes legacy Alloy compose. |
| .yamllint.yaml | Adds repo yamllint config (ignores generated SNMP output). |
| .sops.yaml | Adds SOPS policy template with placeholder recipient. |
| .markdownlint-cli2.yaml | Adds markdownlint-cli2 config. |
| .gitleaksignore | Adds documented ignores for known historical findings. |
| .gitleaks.toml | Adds gitleaks config + homelab-specific rules/allowlist. |
| .gitignore | Replaces gitignore with structured ignores for secrets/runtime artifacts. |
| .github/workflows/ci.yml | Adds CI workflow for lint/validation/secret scanning. |
| .github/pull_request_template.md | Adds PR template aligned with blast radius + verification. |
| .github/ISSUE_TEMPLATE/change.yml | Adds “planned change” issue template. |
| .github/ISSUE_TEMPLATE/bug.yml | Adds “bug” issue template. |
| .github/dependabot.yml | Adds Dependabot config for compose/actions updates. |
| .gitattributes | Marks generated files + normalizes line endings. |
| .editorconfig | Adds editorconfig with Alloy tab indentation settings. |
Suppressed comments (1)
.github/workflows/ci.yml:138
- This second gitleaks scan also uses
zricethezav/gitleaks:latest. For the same reason as the working-tree scan, pin the image to a specific version or digest so full-history scan behavior is deterministic.
run: |
docker run --rm -v "$PWD:/repo" -w /repo \
zricethezav/gitleaks:latest \
detect --no-banner --redact -c .gitleaks.toml --log-opts="--all" -v
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+7
to
+11
| | File | Committed? | Encrypted? | Contains | | ||
| | --- | --- | --- | --- | | ||
| | `observability.example.yaml` | yes | no | Key names and placeholder values | | ||
| | `observability.sops.yaml` | yes | **yes** | Real credentials | | ||
| | `~/.config/sops/age/keys.txt` | **never** | n/a | The private key | |
Comment on lines
+83
to
+84
| envsubst '${SNMP_COMMUNITY_PFSENSE} ${SNMP_COMMUNITY_APC} ${SNMP_COMMUNITY_MOKERLINK} ${SNMP_COMMUNITY_ILO}' \ | ||
| < "${SNMP_SRC}" > "${SNMP_OUT_DIR}/snmp.yaml" |
Comment on lines
+114
to
+117
| -e SNMP_COMMUNITY_MOKERLINK='$${SNMP_COMMUNITY_MOKERLINK}' \ | ||
| -e SNMP_COMMUNITY_ILO='$${SNMP_COMMUNITY_ILO}' \ | ||
| prom/snmp-generator:latest generate \ | ||
| -m /opt/mibs -g /opt/generator.yaml -o /opt/snmp.yaml |
Comment on lines
+129
to
+132
| run: | | ||
| docker run --rm -v "$PWD:/repo" -w /repo \ | ||
| zricethezav/gitleaks:latest \ | ||
| detect --no-git --no-banner --redact -c .gitleaks.toml -v |
…view
CI caught the first problem: amtool rejected the config with 'unsupported
scheme "" for URL'. Alertmanager does not expand environment variables in its
configuration, so `url: $ALERTMANAGER_WEBHOOK_URL` was parsed as a literal
string. The supported mechanism is url_file, read at notify time.
scripts/render-config.sh now writes alertmanager/.rendered/webhook_url and
compose mounts it read-only. Because url_file resolves at notify time rather
than at load time, amtool validates the committed config with no secret present
at all, so CI no longer needs a throwaway webhook URL.
Also replaces envsubst with bash parameter expansion in render-config.sh.
envsubst lives in gettext-base and is not guaranteed on a minimal server
install, so a fresh host would have failed with 'command not found' after
decrypting secrets.
Testing that substitution against a community string containing / & \ and $
surfaced a second bug: bash 5.2 enables patsub_replacement by default, which
makes an unescaped '&' in the replacement expand to the matched text — so a
community containing '&' rendered as the literal placeholder it was meant to
replace. Disabling the option makes the replacement literal; verified that such
a value now survives intact.
Narrows the generated .env to the two values compose actually interpolates, so
each secret now lives in exactly one rendered file rather than being copied into
.env as well. Drops the empty Alertmanager templates: glob.
Review feedback, all of it correct:
* gitleaks ran as :latest in both scan steps, and snmp-generate used
prom/snmp-generator:latest. Both pinned. A scanner that changes underneath
you gives results you cannot reproduce; a generator that changes gives diffs
you cannot explain.
* The 'no floating tags' check only grepped compose.yaml, which is exactly why
those two slipped through — it could not see the file it was defined in. It
now covers every YAML, Makefile and shell script in the repo.
* scripts/validate.sh ran only the working-tree gitleaks scan while CI ran
that plus full history, so `make validate` could pass while CI failed. It
now runs both, which is what "the same set CI runs" was meant to mean.
* secrets/README.md described observability.sops.yaml as committed when it is
absent from the branch. It has to be: encrypting it needs an age keypair,
and generating one here would mean committing a private key. The table now
says so, and notes that `make up` fails loudly until `make secrets-init`
has been run.
CI failed with 'unknown flag: --verify'. Alloy v1.6.1's fmt command takes --write/-w and --test/-t; there is no --verify, and no 'alloy validate' subcommand in this version. Verified against the real v1.6.1 binary rather than by reading the docs a second time: config.alloy is already canonically formatted, 'fmt --test' exits 0, and loading the log_processor block in a real Alloy instance shows the component starting cleanly — which is what actually compiles the regex. That also confirmed the earlier backtick fix does what it was meant to. With the double-quoted string the pattern matched 0 of 4 representative log lines, because \b was a backspace byte; with the backtick string it matches 3 of 4, the fourth having no level token at all. scripts/validate.sh now prefers a local alloy binary over docker, matching how it already handles promtool and amtool, so the whole suite runs without a daemon. Step labels updated to name the flag they actually pass, and both note that fmt checks syntax and formatting but not component configuration.
Gerrrt
added a commit
that referenced
this pull request
Aug 2, 2026
…ing them Dependabot merged version bumps into main while this branch was open, which exposed a design flaw in the CI added in #1: the image pins were duplicated across compose.yaml, ci.yml, validate.sh and check_loki_rules.sh, and Dependabot only updates compose.yaml. The result after those merges: compose.yaml prom/prometheus:v3.13.2 grafana/loki:3.7.4 grafana/alloy:v1.18.0 ci.yml prom/prometheus:v3.1.0 (n/a) grafana/alloy:v1.6.1 scripts prom/prometheus:v3.1.0 grafana/loki:3.3.2 grafana/alloy:v1.6.1 CI was set to validate configs against versions two years older than the ones the stack deploys, and would have reported green while doing it. Validating the wrong version is worse than not validating, because it still produces a passing check. scripts/image-for.sh now reads the pin straight out of compose.yaml, and ci.yml resolves the images into $GITHUB_ENV at run time rather than declaring them. compose.yaml is the single source of truth, so a Dependabot bump automatically reaches every check. The Makefile's snmp-generator is derived from the snmp-exporter pin — the two are released together and the generator is not a compose service, so it cannot be looked up directly. A new CI step fails on any pinned image version outside compose.yaml, so the duplication cannot creep back in. The gitleaks image stays declared in ci.yml because it is a CI tool rather than part of the stack. Verified against the versions actually deployed, not the ones I had lying around: the Loki rules parse and evaluate on 3.7.4 (8/8), loki-config.yaml is valid on 3.7.4, and config.alloy passes fmt --test on v1.18.0 with an empty diff against its canonical formatting — so the formatter did not change behaviour between v1.6.1 and v1.18.0. Also corrects comments that named specific tool versions now resolved dynamically.
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Turns the repository from a config backup into a deployable, self-validating project.
refactor(repo)<service>/<fqdn>/intostacks/observability/fix(observability)feat(secrets)feat(observability)cidocsfix(alertmanager)url_fileinstead of an env var; review fixescialloy fmt --test, the flag that existsWhy
Four defects meant nothing in the repository could actually run, and none were visible without trying:
./prometheus.ymlvs the committedconfig/prometheus.yaml;/opt/loki/loki-config.yml;/opt/alloy/config.alloy.grafana's compose file was invalid. It declareddepends_on: [prometheus, loki], neither defined in that file — compose rejects this outright./grafanainstead of/var/lib/grafana, wiped on every recreate. Loki wrote to/tmp/lokiwhile its volume was at/loki.config.alloy. The log-level regex used\binside a double-quoted Alloy string, where it is a backspace escape rather than a word boundary.Plus three committed credentials, no CI, no dashboards, no alerting.
The Alloy regex, measured
Verified against the real v1.6.1 binary rather than by inspection. Same pattern, same inputs, only the string-literal type differs:
"…")`…`)sshd[811]: error: maximum authentication attempts exceedederrorlevel=warn msg="disk nearly full"warnCRITICAL: power failure detectedCRITICALnothing interesting happened here0/4 → 3/4. Every log line was falling through the
{{ else }}info{{ end }}branch, so{level="error"}returned nothing, ever.Blast radius
snmp-exportermoves to compose-internal only, Alloy's UI binds to127.0.0.1secrets/*.sops.yamlDeploying this is not a no-op on the running host: service names replace hardcoded IPs, and Grafana's and Loki's data paths both change. Follow
docs/runbooks/deploy-stack.md.Security
Three credentials were committed to this public repository and remain in git history:
admin/adminwithGF_AUTH_ANONYMOUS_ENABLED=trueand Admin rolecertificates/scripts/purge-history.shand a runbook are providedRotating live credentials and rewriting history both need access this branch does not have, so they ship as a tested script plus runbooks. Both are tracked in
docs/roadmap.mdand documented indocs/security.md..gitleaksignoreenumerates all nine historical findings with an explanation of each. It is an acknowledgement, not a fix — a CI job permanently red for a known reason gets ignored, and then a genuinely new leak goes unnoticed alongside it.docs/network.mdpreviously published the full MAC, OS version and room of every device in the house, including cameras and a baby monitor. MACs are now truncated to their OUI, personal devices listed by role, camera-to-room mapping removed.Verification
CI is green.
make validateruns the same checks locally and passes end-to-end against realpromtool,amtool,alloy,gitleaksandshellcheckbinaries:Beyond the linters:
log_processorcomponent was loaded in a real Alloy instance, which is what compiles the regex; it starts cleanly.p/f&s\en$e!1). This caught a bug: bash 5.2 enablespatsub_replacementby default, so an unescaped&in a replacement expands to the matched text — a community string containing&would have silently rendered as the literal${SNMP_COMMUNITY_PFSENSE}placeholder. Fixed and re-verified byte-for-byte.Not verified — no Docker daemon was available in the authoring environment:
docs/images/README.mddocuments how to capture and sanitise them; the README uses Mermaid rather than placeholder images.make secrets-initgenerates it; pre-seeding it would have meant committing a private key.Follow-up
docs/runbooks/rotate-snmp-community.md.gitleaksignore—docs/runbooks/purge-git-history.mdups.rules.yamlcurrently reports on a UPS that cannot hold the load)ifXTableto the switch SNMP module — the 32-bit counters wrap in ~34s at gigabit line rate