feat: docker compose observability stack

[fatih-acar]

This PR updates the observability stack shipped in the development/ folder (should we move stuff out of this folder?) - drops Prometheus for scraping and Promtail, replaced by Grafana Alloy.

PR also includes a script that embeds all the file based configs within docker-compose configs so it can be launched in through a self contained docker compose file.

This is INFP-466

---

Summary by cubic

Replaces Promtail and manual Prometheus scrapes with grafana/alloy, and adds a single-file observability Docker Compose for easy local deployment. Pins image versions, adds a CI check to keep the standalone file in sync, and updates docs for enabling observability and tracing (INFP-466).

• New Features

  • Added grafana/alloy to collect logs and scrape metrics; logs go to Loki and metrics are remote-written to Prometheus. Filters containers by COMPOSE_PROJECT_NAME and exposes self-metrics on port 12345.
  • Introduced convert_compose_standalone.py to embed configs and generate docker-compose-observability-standalone.yml; CI regenerates and validates it’s up to date.
  • Updated docs with ?observability=true, tracing env vars, and a --force-recreate note for config changes.

• Dependencies

  • Replaced grafana/promtail with grafana/alloy:v1.15.0; bumped grafana/loki to 3.7.1, grafana/tempo to 2.10.3, prom/prometheus to v3.11.0, prom/node-exporter to v1.10.2, gcr.io/cadvisor/cadvisor to v0.52.1, prefecthq/prometheus-prefect-exporter to 3.3.0.
  • Removed development/prometheus/config.yml and development/promtail/config.yml; Prometheus now runs with an empty config and only receives remote write.

^{Written for commit 2224d07. Summary will update on new commits.}

[codspeed-hq]

Merging this PR will not alter performance

✅ 12 untouched benchmarks

---

_{Comparing fac-INFP-466-docker-compose (10aa85f) with stable (1ac7194)}

[cubic-dev-ai]

2 issues found across 9 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="development/alloy/config.alloy">

<violation number="1" location="development/alloy/config.alloy:179">
P1: `source = "tmp_msg"` references an extracted field that is never populated by any prior stage in the pipeline. Per the Alloy docs, when `source` is set, it parses that previously extracted value — since `tmp_msg` never exists, this entire `stage.logfmt` block is a no-op and the `app`, `request_id`, `branch`, and `trace_id` fields will never be extracted from non-JSON logs.

If the intent is to parse the raw log line, remove the `source` parameter (empty/missing means "parse the log line itself"). If the intent is to parse a specific extracted field, a prior stage needs to populate `tmp_msg`.</violation>

<violation number="2" location="development/alloy/config.alloy:380">
P2: `job_name = "task-worker"` is misleading for a scrape targeting `prefect-server:4200/api/metrics`. This likely should be `"prefect"` or `"prefect-server"` to match the actual target. The current name would cause confusion in dashboards and alerting since it doesn't describe the Prefect server being scraped.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review, or fix all with cubic.}