Draft
Conversation
Replace the single-purpose upload.py with a unified CLI that supports both `add-version` and `check` subcommands, with `--target s3` and `--target local:<path>` for easy local development and dry runs. Signed-off-by: Claude <noreply@anthropic.com> https: //claude.ai/code/session_012GN6N8fzLnvnCAZa7ozPQj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Documents the subcommands (add-version, check, list), target types, manifest merging rules, CI integration, and example workflows. Signed-off-by: Claude <noreply@anthropic.com> https: //claude.ai/code/session_012GN6N8fzLnvnCAZa7ozPQj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Key changes: - Split add-version into generate (local-only) and upload (to any store) - Store abstraction treats local paths and s3://bucket identically - --store replaces --target, defaults to s3://vortex-compat-fixtures - Manifest merging moves to upload (not generate) - Add list subcommand for inspecting stores Signed-off-by: Claude <noreply@anthropic.com> https: //claude.ai/code/session_012GN6N8fzLnvnCAZa7ozPQj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Consolidates compat-gen, compat-validate, and compat.py into one vortex-compat binary with generate/check/list subcommands and a FixtureStore trait that abstracts local dirs and S3 identically. Signed-off-by: Claude <noreply@anthropic.com> https: //claude.ai/code/session_012GN6N8fzLnvnCAZa7ozPQj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Replace compat-gen + compat-validate binaries and compat.py script with a single vortex-compat binary with four subcommands: - generate: build fixtures and write to any store - check: validate fixtures from any store - list: inspect store contents - validate-manifest: verify additive-only invariant across versions FixtureStore trait abstracts local dirs and S3 identically. S3 reads use public HTTPS (reqwest), writes shell out to aws CLI. Signed-off-by: Claude <noreply@anthropic.com> https: //claude.ai/code/session_012GN6N8fzLnvnCAZa7ozPQj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…h command - Change Fixture::build() to take a tmp_dir path instead of download/build split - Each fixture handles its own downloads using tmp_dir as scratch space - Simplify generate.rs to a two-phase pipeline: parallel build → write - Add publish.rs for generate + upload to store with parallel uploads - Add Send+Sync bounds to FixtureStore trait for thread safety - Add List, ValidateManifest, Publish CLI subcommands - Use parking_lot::Mutex instead of std::sync::Mutex per project conventions Signed-off-by: Claude <noreply@anthropic.com> https: //claude.ai/code/session_012GN6N8fzLnvnCAZa7ozPQj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
- Add `description(&self) -> &str` to every fixture for human-readable docs - Add optional `setup(&self, tmp_dir: &Path)` phase for I/O (downloads) - Run setup concurrently via tokio::spawn_blocking before parallel build - Move ClickBench download from build() to setup() for proper phase separation - Generate pipeline is now: (1) async setup, (2) parallel build, (3) write Signed-off-by: Claude <noreply@anthropic.com> https: //claude.ai/code/session_012GN6N8fzLnvnCAZa7ozPQj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Rust binary (vortex-compat) now has only two commands: - `generate --output <dir>`: build all fixtures, write .vortex files + fixtures.json - `check --dir <dir> --mode exact|subset|superset`: compare files against in-memory Python script (compat.py) handles everything version-agnostic: - S3 uploads/downloads, manifest merging, version tracking, additive-only checks - Calls vortex-compat as a subprocess, parses JSON output This means only the small Rust binary needs porting across Vortex versions, while the Python orchestrator works unchanged for all versions. Removed from Rust: store.rs, publish.rs, validate.rs, manifest.rs, all S3/ETag/manifest logic, ~500 lines of orchestration code. Signed-off-by: Claude <noreply@anthropic.com> https: //claude.ai/code/session_012GN6N8fzLnvnCAZa7ozPQj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…eneration Rust binary: - Added long_about with output protocol description - Added after_help with usage examples - Added value_name annotations for all args - Added detailed doc comments on each check mode Python orchestrator: - Added EPILOG with environment variables, store spec docs, and examples - Each subcommand has its own description + epilog with examples - Uses RawDescriptionHelpFormatter for proper formatting - Added --git-ref to generate and publish commands for worktree-based builds - Added publish-multi command for bulk version publishing with parallel support - Refactored helpers with _ prefix naming convention Updated CLI_SPEC.md to document the new two-layer architecture. Signed-off-by: Claude <noreply@anthropic.com> https: //claude.ai/code/session_012GN6N8fzLnvnCAZa7ozPQj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The publish --git-ref flag already supports worktree-based generation for individual versions. publish-multi can be replaced by a simple shell loop, so it's unnecessary complexity. Signed-off-by: Claude <noreply@anthropic.com> https: //claude.ai/code/session_012GN6N8fzLnvnCAZa7ozPQj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Version is now always derived from the nearest git tag via `git describe --tags --abbrev=0`, stripping the 'v' prefix. - `publish` and `generate` no longer take `--version` - `--git-ref v0.62.0` both builds from that ref AND uses 0.62.0 as version - CI workflow changed from version input to optional git_ref input - fetch-depth: 0 added so tags are available Signed-off-by: Claude <noreply@anthropic.com> https: //claude.ai/code/session_012GN6N8fzLnvnCAZa7ozPQj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Allows excluding specific fixtures by name (comma-separated) from both generation and checking. Useful for skipping fixtures with external dependencies (e.g. clickbench) in environments without network access. Wired through both the Rust binary and Python orchestrator for generate, publish, and check commands. Signed-off-by: Claude <noreply@anthropic.com> https: //claude.ai/code/session_012GN6N8fzLnvnCAZa7ozPQj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Documents the new store format where fixture directories are keyed by
`v{version}-{short_hash}`, versions.json is a flat version→hash map,
and per-version manifests include provenance (commit, dirty, rust_version)
and per-fixture sha256 hashes for oracle drift detection.
Signed-off-by: Claude <noreply@anthropic.com>
https: //claude.ai/code/session_012GN6N8fzLnvnCAZa7ozPQj
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
- Add expected_encodings field to per-fixture manifest with array:/layout: prefix convention and subset assertion semantics - Replace incorrect oracle drift framing for sha256 with integrity check and skip-unchanged-uploads optimization - Add fixture evolution section codifying append-only contract - Document ExpectedEncoding enum and verification at generate/check time Signed-off-by: Claude <noreply@anthropic.com> https: //claude.ai/code/session_012GN6N8fzLnvnCAZa7ozPQj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Narrative design doc covering the problem, fixture contract, oracle approach, expected_encodings verification, architecture split (Rust binary vs Python orchestrator), store format with commit-keyed directories, and known limitations. Signed-off-by: Claude <noreply@anthropic.com> https: //claude.ai/code/session_012GN6N8fzLnvnCAZa7ozPQj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
- Rewrote DESIGN.md as the single source of truth, matching actual code
(compat.py uses `publish`/`check`/`--store`, not `add-version`/`--target`;
store uses `v{version}/` directories, not `v{version}-{hash}/`;
`versions.json` is a flat list, not a map)
- Deleted STORE_FORMAT.md (content absorbed into DESIGN.md store section)
- Deleted CLI_SPEC.md (content absorbed into DESIGN.md orchestrator section)
- Rewrote README.md as a quickstart with AWS setup, pointing to DESIGN.md
Signed-off-by: Claude <noreply@anthropic.com>
https: //claude.ai/code/session_012GN6N8fzLnvnCAZa7ozPQj
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
- Add ExpectedEncoding enum (Array/Layout variants) with Display impl producing "array:vortex.foo" / "layout:vortex.foo" format - Add expected_encodings() to the Fixture trait - Implement expected_encodings() on all 9 fixtures, declaring the array and layout encodings each is designed to exercise - Include expected_encodings in fixtures.json output from generate - Pass expected_encodings through in compat.py manifest merging Signed-off-by: Claude <noreply@anthropic.com> https: //claude.ai/code/session_012GN6N8fzLnvnCAZa7ozPQj Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The --exclude flag previously required exact fixture names (e.g. "clickbench_hits_1k.vortex"). Now it matches substrings, so "--exclude clickbench" correctly excludes clickbench_hits_1k.vortex. Validated full round-trip: generate -> check with all 6 synthetic fixtures passing in both exact and subset modes. Signed-off-by: Claude <noreply@anthropic.com> https: //claude.ai/code/session_012GN6N8fzLnvnCAZa7ozPQj Signed-off-by: Claude <noreply@anthropic.com>
joseph-isaacs
added a commit
that referenced
this pull request
Mar 17, 2026
Add extract fixtures to the backtesting system. There is another PR that will change the way we actually write the file to a store [see #6993]. Focus only on the generated files and the data from each fixture. --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes: #000
Testing