chore(oo): single instance per host and drop Vector self-metrics - #3004
Merged
Conversation
oo1.prosopo.io and oo2.prosopo.io are separate hosts, but the oo1 compose file was running both containers on the same host with a shared /data/oo bind mount, so two openobserve processes were racing each other on the same WAL and parquet tree. That doubled CPU and disk writes on tiger-stairs, and Caddy's fallback upstream targeted the wrong port (5081 is the host-side mapping) so oo2 never actually served a request anyway. Also stop shipping Vector's internal_metrics to OpenObserve. It fans every histogram bucket into a separate stream and was responsible for ~1,000 metric streams that nothing dashboards on, dominating the metastore size and per-flush compaction work. - Remove the oo2 service from docker-compose.oo1.yml - Simplify oo1.Caddyfile reverse_proxy to a single upstream - Drop [sources.vector_metrics] + [transforms.vector_metrics_format] and remove them from both prometheus_remote_write sinks Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
oo1.prosopo.ioandoo2.prosopo.ioare separate machines, butdocker-compose.oo1.ymlwas starting both anoo1and anoo2container on the same host, sharing/data/oo. Two openobserve processes were racing each other over the same WAL and parquet tree, doubling CPU and disk writes on tiger-stairs. Caddy's fallback upstream also targetedoo2:5081, but 5081 is the host-side mapping — the container listens on 5080 internally, so the fallback never actually worked.oo1.Caddyfilereverse_proxyto a single upstream (oo1:5080), nolb_policyblock.internal_metricssource + transform fromvector.toml, and strippedvector_metrics_formatfrom bothprometheus_remote_writesinkinputsarrays. Vector fans every histogram bucket into its own Prometheus series → its own OpenObserve stream; this was responsible for ~1,000 low-value metric streams that dominated metastore size and compaction cost.Only Vector's metrics pipeline is affected. All log pipelines (provider, mongo, caddy, redis, docker events, Vector's own
internal_logs) are untouched. Host / cAdvisor / Redis / Mongo / Caddy exporter metrics still ship as before.Test plan
docker compose -f captcha/docker/docker-compose.oo1.yml --profile production configparses cleanlyoo1+caddycome up;oo.prosopo.iostill serves the UI and ingest endpoints*_provider_*streams in OpenObservevector_*metric streams created after the Vector image is rolled out to a provider🤖 Generated with Claude Code