Add multi-endpoint recording support#793
Merged
thinkingfish merged 4 commits intoiopsystems:mainfrom Apr 17, 2026
Merged
Conversation
thinkingfish
force-pushed
the
feat/multi-endpoint-recorder
branch
from
80c143c to
5b03439
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
thinkingfish
force-pushed
the
feat/multi-endpoint-recorder
branch
from
5b03439 to
dae4ef2
Compare
Design for extending `rezolus record` to scrape multiple endpoints concurrently with mixed protocol support (msgpack + Prometheus), combined/separate output modes, late-joiner handling, and per-source column metadata provenance. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Support concurrent scraping from multiple endpoints with mixed protocols (Rezolus msgpack and Prometheus text). Adds three CLI modes: positional URL (backward compat), --endpoint url,source=name (repeatable), and --config file.toml. Includes protocol auto-detection, late-joiner retry, per-source column metadata provenance, and combined/separate output modes. Key changes: - New endpoint.rs: Protocol, EndpointConfig, EndpointState types - New config.rs: TOML config parsing, unified RecordingConfig - Rewritten recorder run() with concurrent fan-out via join_all - PrometheusConverter gains source/endpoint provenance injection - inject_provenance() for msgpack snapshots (V1 and V2) - Separate output mode (--separate) writes per-endpoint parquet files - Combined output reuses parquet_tools::combine logic - Per-source metadata: first/last sample timestamps, role Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
thinkingfish
force-pushed
the
feat/multi-endpoint-recorder
branch
from
adaf0f3 to
356b591
Compare
Upstream added a bypass_time_check parameter to combine_and_write. The recorder's combine_files path always validates, so pass false. Also remove unused std::str::FromStr import.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
rezolus record<URL> <OUTPUT>(backward compat),--endpoint url,source=name(repeatable),--config file.tomlsource/endpointinjected into every metric)--separatewrites per-endpoint filesfirst_sample_ns/last_sample_nstimestamps and optionalroleKey files
src/recorder/endpoint.rs— Protocol, EndpointConfig, EndpointState typessrc/recorder/config.rs— TOML config parsing, unified RecordingConfig, CLI arg handlingsrc/recorder/mod.rs— Rewritten run() with concurrent fan-out, probe, provenance injectionsrc/recorder/prometheus.rs— PrometheusConverter provenance supportTest plan
cargo test— 103 tests passcargo fmt— cleanrezolus record <URL> <OUTPUT>backward compatrezolus record --endpoint url1,source=a --endpoint url2,source=b,protocol=prometheus out.parquetrezolus record --config recording.toml--separatemode produces per-endpoint files🤖 Generated with Claude Code