Add broker_source label to all Prometheus metrics

[jghoman]

Summary

Add a broker_source label to every Prometheus metric, enabling filtering by broker type (e.g. msk vs warpstream) in Grafana during the ingestion migration to WarpStream.

Changes

• config.py: New broker_source field, read from BROKER_SOURCE env var (default empty)
• metrics.py: broker_source added as a label to all metric definitions. _AutoPipelineLabels renamed to _AutoCommonLabels to inject both pipeline and broker_source.
• main.py: Passes broker_source to metrics.init()
• tests/unit/test_consumer.py: Fixture updated for new config field

Deployment

Requires companion charts PR (PostHog/charts) to set BROKER_SOURCE env var:

• MSK consumers: BROKER_SOURCE=msk
• WarpStream consumers: BROKER_SOURCE=warpstream

Backward compatible — if BROKER_SOURCE is unset, the label defaults to empty string.

[Copilot]

Pull request overview

Adds a broker_source Prometheus label across millpond metrics to support Grafana filtering during the Kafka ingestion migration (e.g., distinguishing MSK vs WarpStream consumers).

Changes:

• Add broker_source to Config, sourced from BROKER_SOURCE env var (default "").
• Add broker_source label to all Prometheus metric definitions and update metrics initialization to bind both pipeline and broker_source.
• Wire broker_source through main() into metrics.init(), and update unit test config fixture.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
millpond/config.py	Adds broker_source config field loaded from BROKER_SOURCE.
millpond/metrics.py	Adds broker_source label to all metrics and binds it during init().
millpond/main.py	Passes cfg.broker_source into metrics.init().
tests/unit/test_consumer.py	Updates Config fixture to include broker_source.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.