Skip to content

test(odb): establish dynamic store correctness and performance baselines - #2853

Draft
Sebastian Thiel (Byron) wants to merge 10 commits into
mainfrom
odb-parallelism
Draft

test(odb): establish dynamic store correctness and performance baselines#2853
Sebastian Thiel (Byron) wants to merge 10 commits into
mainfrom
odb-parallelism

Conversation

@Byron

@Byron Sebastian Thiel (Byron) commented Jul 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Tasks

  • refackiew
  • ensure there is no performance regression (around parallelism)

Codex inserts after this line

Purpose

This PR establishes the correctness and performance baseline for upcoming work on the multi-handle, multi-threaded dynamic object database. It intentionally changes test infrastructure rather than production ODB behavior: the aim is to make filesystem transitions, refresh behavior, slot pressure, and handle interactions reproducible before changing the implementation.

The infrastructure is intended to support this sequence of follow-up work:

  1. Remove stray yields.
  2. Remove loading races, propagate the winning load error correctly, and avoid repeatedly attempting the same failing file.
  3. Allow the slot map to grow dynamically, which can also remove the need to scan the packs directory when opening a repository.
  4. Avoid refresh races and unnecessary refreshes after misses while retaining configurable correctness bounds: strict/current behavior, a tolerant freshness window, and Refresh::Never.

What this adds

Shared ODB scenarios

A gix-testtools fixture script builds a deterministic catalog of valid SHA-1 and SHA-256 packs and indices. A utility module under gix-odb/tests/ copies and mutates that catalog for tests, benchmarks, and fuzzing without introducing another support crate.

The shared vocabulary can:

  • publish or remove pack, index, reverse-index, bitmap, promisor, mtimes, and keep components independently;
  • create and replace multi-pack indices;
  • add and remove alternates;
  • corrupt and restore indices;
  • model complete valid states as well as temporary states produced by Git-style maintenance.

This keeps all three kinds of coverage grounded in the same well-known repositories and object IDs.

Deterministic correctness coverage

The integration tests exercise multiple live handles in deliberately interleaved, single-threaded sequences while:

  • packs are published and removed component by component;
  • MIDX files are rewritten and referenced packs disappear;
  • alternates change;
  • malformed indices are restored;
  • fixed slots are over-committed.

The existing threaded packed-object smoke test also now always uses at least two workers and verifies that every enumerated object is actually found.

Benchmarks

Criterion benchmarks provide baselines for:

  • opening with a pack-directory scan versus a fixed slot count;
  • cold and warm hits through standalone indices and a MIDX;
  • repeated misses with default refresh behavior versus Refresh::Never;
  • bulk prefix disambiguation with both refresh modes;
  • concurrent hits and misses at one worker, available parallelism, and twice available parallelism.

These should make regressions and improvements in refresh suppression, prefix lookup, startup behavior, and slot over-commitment visible.

Deterministic fuzz VM

A single-threaded bytecode VM interleaves operations across several handles with disk mutations. It covers lookup, header access, iteration, prefix queries, refresh modes, pack-component transitions, MIDX transitions, alternates, malformed indices, and slot pressure.

The seed corpus includes standalone packs, MIDX use, alternates, malformed recovery, slot pressure, pack replacement, MIDX expiration, and interrupted maintenance. Maintenance states can expose markers or new components before old packs are removed, remove packs while a MIDX still references them, and abandon or reverse transitions.

The VM checks returned object hashes, header/data agreement, fresh-store equivalence at valid checkpoints, and existing handle metrics. Store statistics can be used or extended further where they help characterize later correctness or performance changes.

Verification

  • cargo test -p gix-odb --all-features --lib --test odb
  • SHA-256 runs of the new dynamic scenarios
  • cargo clippy -p gix-odb --all-features --tests --benches -- -D warnings
  • Criterion benchmark smoke run
  • nightly fuzz build, fixed-seed replay, and a short generated-input run
  • shellcheck for both fixture scripts

@Byron Sebastian Thiel (Byron) changed the title improve odb correctness test(odb): establish dynamic store correctness and performance baselines Jul 30, 2026
@Byron
Sebastian Thiel (Byron) force-pushed the odb-parallelism branch 6 times, most recently from c6a213f to 8047243 Compare July 30, 2026 20:18
@Byron
test(gix-odb): establish reusable dynamic-store scenarios
124c9e4 
Add a gix-testtools-generated SHA-1/SHA-256 ODB catalogue and a shared mutation helper for publishing and removing pack components, rewriting MIDX files, changing alternates, and constructing maintenance-like intermediate states.

Exercise deterministic single-threaded interactions across multiple handles, malformed-index recovery, changing packs and alternates, MIDX transitions, and slot over-commitment. This establishes the ordinary regression vocabulary used by the subsequent implementation changes, benchmarks, and fuzz model.
@Byron
perf(gix-odb): establish dynamic-store baseline workloads
c2949d1 
Use the shared deterministic ODB fixture to measure startup scanning, cold and warm packed lookups, repeated misses, prefix disambiguation, pack publication, fixed-slot pressure, concurrent access, and CPU over-commitment before changing the dynamic ODB implementation.

Compare the original strict and never-refresh policies, and use Store metrics outside timed paths to verify that each workload reaches its intended refresh behavior. These common benchmark groups remain available after the implementation changes for direct Criterion comparisons.
@Byron
fix(gix-odb): reconcile changing pack state without losing usable data
8758c57 
Make disk reconciliation atomic when slots are exhausted or reused, retry changed indices after a cached load failure, and remember unchanged pack-load failures without repeatedly reopening the same file.

Record an attempted lazy initialization for never-refresh handles so misses do not rescan forever, and recover unexpectedly vacant recorded slots through normal reassignment instead of assuming they are unreachable. Existing handles retain their last usable state throughout failed or partial maintenance transitions.
@Byron
change!: rename Slots::Given to Slots::Limit
08ef674 
The fixed-size slot mode is a hard upper bound: once all slots are occupied,
new disk state yields InsufficientSlots. Calling it Given obscures that behavior,
especially now that a separate mode will be able to grow.

Rename the public variant and update all workspace callers and documentation.
This commit contains only the breaking vocabulary change so downstream
adaptation is easy to identify.
@Byron
feat: grow the dynamic ODB slot map on demand
1c1e667 
Opening a store should not require listing every pack merely to size a fixed slot map, and later maintenance should not fail only because the repository outgrew its opening-time estimate.

Add a Growable slot mode whose initial allocation performs no disk scan. Grow by publishing copy-on-write snapshots of stable slot pointers under the existing structural lock, so readers remain lock-free and all prior slot IDs stay valid. AsNeededByDiskState also gains growth after its initial estimate, while Limit retains the former hard-cap behavior.

Ordinary tests cover growth past the initial allocation with shared and stable handles.
@Byron
fix(gix-odb): publish maintenance-safe store state atomically
5d5fd52 
Publish the slot index and its backing resource nodes through one catalog so handles cannot combine refresh generations or lazily load a pack through a reassigned slot. Snapshots and stable locations retain the exact index and pack resources they observed, without Store-local PackId encoding or artificial slot/member bit limits.

Recover republished indices and packs only when their on-disk state actually changed, preserve loaded MIDX packs across redundant loads, and replace rewritten MIDX resources at their logical slot even at a configured limit.

Add test-only synchronization hooks and make contending index loads, pack loads, and refreshes share their success or error outcome. This prevents duplicate I/O and keeps failed maintenance transitions from partially replacing usable state.
@Byron
feat(gix): throttle object refreshes with a freshness window
f89979e 
Add RefreshMode::AfterDuration as a shared gix-odb freshness window. Repeated misses avoid rescanning packs and alternates until the last successful post-initialization refresh reaches the configured age, while failed refreshes, known missing packs, and explicit stale notification remain immediately retryable.

Configure gix object handles with a one-second default and expose gitoxide.objects.refreshAfter in milliseconds. Fetch invalidates freshness after publishing a pack, configuration reloads update active handles, and zero restores strict refresh-on-every-miss behavior.

Use the test-support clock for deterministic deadline and contention coverage, and keep feature-unified workspace builds and Repository size guards valid with the added configuration.
@Byron
fuzz(gix-odb): model dynamic-store operations and maintenance
e7525de 
Add one deterministic single-threaded opcode VM for multiple handles, component-wise pack publication and removal, MIDX replacement, alternates, malformed resources, loose-object lifecycle, slot pressure, stable locations, iteration, prefix operations, and refresh policies.

Give every opcode an explicit reproducible outcome and validate queries, shared-store checkpoints, retained locations, refresh counts, and fixture convergence against the generated disk model. Deterministic time covers strict, duration-based, and never-refresh behavior without wall-clock races.

Keep readable corpus seeds for interrupted maintenance, load recovery, stale snapshots, cloned handles, MIDX transitions, growable stores, and minimized regressions.
@Byron
perf(gix-odb): extend benchmarks for growth and freshness policies
4098758 
Extend the pre-change dynamic-store baseline with scan-free growable opening, first-growth cost, fresh and expired duration policies, and post-publication discovery through an expired freshness window.

Retain the common strict and never-refresh benchmark groups unchanged so Criterion results remain directly comparable across the implementation boundary, while the additional cases measure behavior that did not exist in the baseline API.
@Byron
fix CI dependency check without ripgrep
af4b335 
The minimal Linux test job does not install ripgrep. Its inverted pipeline therefore succeeded because rg was missing, silently skipping the assertion that a no-default-features gix build excludes gix-imara-diff.

Use POSIX grep with extended regular expressions, which is already required by neighboring checks and preserves the exact dependency assertion on every CI image.
@Byron Sebastian Thiel (Byron) linked an issue Aug 5, 2026 that may be closed by this pull request
gix-odb: Handle::contains issues a sched_yield on every miss, including under RefreshMode::Never #2878
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

gix-odb: Handle::contains issues a sched_yield on every miss, including under RefreshMode::Never

1 participant

@Byron