Repository navigation
Releases: itshak/cbvault
Release list
v0.1.6
Fixed
- Unindexed position search honours cancel.
for_each_position_keyand
for_each_position_key_rangeaskedPositionQuery::cancel_withonly after
every chunk had been replayed, so a cancelled search still walked the whole
set (minutes on the Mega Database) before reportingcomplete = false. The
cancel is now asked before each chunk and every 256 games within one; a
cancelled run replays no further games. Found when BlindBase moved its
unindexed.cbhexplorer onto this API with the early-stop predicate, and a
superseded query kept every core busy.
Full Changelog: v0.1.5...v0.1.6
v0.1.5
Sorted key tables for namebases with prefix lookups, early-stop replay pruning,
pipelined parallel conversion with an ordered reorder buffer, and gigachess 0.1.10.
Added
-
Namebase sorted key table (
KeyTable). Replaced the$O(N)$ tree walk in
Namebase::find_allwith an in-memory flat sorted key table built on first lookup.
For ~1M players, builds in < 300 ms and gives$O(\log N)$ binary search lookup
(under 5 ms warm), completely resilient against damaged binary tree links in the file. -
Prefix lookup on namebases. Added
Entities::find_players_prefix,
find_tournaments_prefix,find_annotators_prefix,find_sources_prefix, and
find_teams_prefixfor fast prefix searches. -
Early-stop walk predicate (
Flow). AddedFlow::{Continue, SkipGame, Stop}
andPositionQuery::early_stop_with, allowing per-game pruning during unindexed
position search (e.g. viagigachess::reach::ReachSignature). -
Parallel conversion walk with ordered reorder buffer. Pipelined chunk decoding
and delivery through a bounded reorder buffer, eliminating wave-barrier synchronization
while preserving strict output order, deduplication, and recycling chunk allocations. -
Dependency bump: Upgraded
gigachessto0.1.10.
Full Changelog: v0.1.4...v0.1.5
v0.1.4
Cooperative walk cancellation and one PGN game splitter, both driven by the
BlindBase live-set position search: a 50-game answer over an 11M-game set no
longer costs a full decode.
Added
GameSink::cancelled(defaultfalse), polled per record by
for_each_game/for_each_rangeand per wave byconvert_parallel.
A cancelled run returns its prefix stats withConvertStats::complete = false,
mirroringPositionSearch::complete— a prefix is never mistaken for a whole answer.cbvault::pgn::split(split_games,split_game_ranges): the one
boundary rule (blank line outside a comment whose next non-blank line is a
tag pair), byte-based with lossy UTF-8, ported from BlindBase with its test
vectors. The rule has one owner now so both sides of the bridge split the
same bytes the same way.
Full Changelog: v0.1.3...v0.1.4
v0.1.3
Direct PGN extraction on Database façade, high-performance binary-search multi-player filtering (Filter::PlayerSet), and 0-ply score-only game contract.
Added
-
Database::game_pgn&Database::game_pgn_with, generating complete, valid PGN strings (Seven-Tag Roster, optional event/site/eco/round metadata, and formatted move text) directly from raw.cbhand.cbgbytes without requiring consumer-side move re-encoding. -
Filter::PlayerSet(IdSet), providing zero-allocation$O(\log K)$ binary-search matching over sorted candidate player IDs for multi-name queries. -
Score-only (0-ply) game contract, ensuring games without move bytes (e.g. historical games like Staunton–Hughes 1858) emit valid PGN headers with
[PlyCount "0"]matching official ChessBase export behavior instead of silently skipping them.
Full Changelog: v0.1.2...v0.1.3
v0.1.2
Search vocabulary for a raw ChessBase database, so a consumer can express the
query its UI actually sends.
Added
- Rating ranges, and the other header predicates a search form needs.
FiltergainsWhiteEloBetween,BlackEloBetween,EloBetween,
YearBetween,Result,RoundandEco, over aRangetype whose absent
end is open rather than zero. A threshold is not a range: "rating from
2800" and "rating at least 2800" are different questions, and mapping one onto
the other either loses the upper bound or invents one.Range::at_least
makes a threshold a special case of a range, so both answer through one
comparison.Filter::eco_text("B20")parses the text form a UI holds and
returnsNonefor text that is not a code, so a typo can be reported rather
than becoming a predicate that matches nothing. Entities::find_players/find_tournaments/find_annotators/
find_sources/find_teams, which resolve a name to every id it names.
.cbpis keyed by last name, so one key can have many records; the existing
find_playeranswers "an id", and a filter built from one silently omits
every other player's games — which reads to a user as "this player has fewer
games than I thought" rather than as a bug.find_playeris kept for a caller
that wants only an answer.AllOf, a conjunction builder, andany_player, which turns a
resolved id set into one filter. An empty conjunction matches every game, so
clearing a consumer's search form does not become a zero-result query; an
empty id set matches nothing, so a misspelt name does not return the whole
database. Those two are opposites on purpose, and both are cheap to get
backwards.GameRef::plies()andhas_keys(). The ply count ismoves.len()and
the walk already knows it, so it is free. It is named because the obvious
shortcut is wrong: the header'smove_count()is au8capped at 255 and
counts moves, so2 * move_count()is wrong for any game with a set-up
start, and silently so.
Fixed
- The test fixtures wrote the namebase tree root at the wrong header offset,
and wrote no tree at all. The reader reads the root at0x04, the second
header field; the fixture wrote it fifth. Every fixture namebase therefore had
a root of the literal0— a valid-looking leaf — so a namebase lookup searched
record 0 and returned a plausible answer. Nothing noticed, because record 0 is
often the right one. Separately, every child link was-1, so the tree was a
single node and a reader walking it found one record however many existed. The
tree-descent path was therefore never exercised by a fixture. Fixtures now
write a balanced tree over the name fields, and the root lands at the offset
the reader reads. find_all— the new multi-id lookup — did not terminate. The tidy
three-push in-order walk re-pushes each node, which puts its left child on the
stack a second time and that child's left child on again; a seven-record
fixture spent its whole budget popping one leaf and returned a single id. It is
a pre-order walk with aseenset, which is also correct on a damaged tree
rather than merely bounded.
Not changed
- No new traversal. The plan assumed this release would add a key-emitting
walk with pluggable sinks. It does not need to:GameSinkalready receives a
GameRefcarrying the record number, themoves2main line and one Polyglot
key per position, driven byfor_each_gameandconvert_parallel, with
wants_keysas the make-selection contract ADR-003 moved up a level. A
consumer's index build, opening tree and position search become three sinks on
one existing walk. - Additive API only. No existing item changed behaviour or signature, and
nothing here decodes a move differently, so the byte-equality gates
(ADR-002, ADR-003) are untouched.
Full Changelog: v0.1.1...v0.1.2
v0.1.1
A build fix, and a CI change that is the more important half of it.
Fixed
cbvault-formatdid not compile on Windows.DbFile::opencalled
memmap2::Advicebehind the crate's default-onmmapfeature, gated on the
feature and not on the target.AdviceandMmap::adviseare#[cfg(unix)]in
memmap2, so on Windows the crate failed withE0433: cannot find Advice in memmap2andE0599: no method named advise found for &Mmapand nothing
depending on it could be built. Found by BlindBase'sv0.4.3release build on
windows-x64; the same release'smacos-arm64job passed. The advisory
Sequentialhint is now gated#[cfg(unix)], so wherememmap2offers no
equivalent the mapping is used as it is. No decoding, API, error or output
change on any platform, and no change at all on Unix.DbFile::reopenwas lint-dirty. Windows-only, and — with no Windows job in
CI — never compiled by anything. Itsunsafe extern "system"block and two
unsafeblocks now carry#[allow(unsafe_code)](each already has aSAFETY
comment), and the///on the extern block, which rustdoc does not document, is
a plain comment. Four warnings, zero now.
Changed
- CI runs on
windows-latest. Both operating systems were Unix, which is why
a green tree published a version that did not compile on one of the two desktop
platforms its only consumer ships: a Unix-only matrix cannot see a Unix-only
API, and every#[cfg(windows)]line in this crate was unbuilt by CI. The
Windows job compiles all targets, lints with-D warnings, runs the suite and
compiles the benchmarks, like the other two. - The CI job's steps run under bash on all three platforms. The toolchain
check uses bash parameter expansion andsort -V; PowerShell is the Windows
default and would have failed the job for a reason unrelated to the code.
Not changed
- No format, decoding path, public API, error type or output byte.
- No dependency added, removed or upgraded.
- No benchmark-affecting change: the dropped call is an advisory hint, and it is
dropped only where it does not exist.
Provenance
No new module and no third-party material. The two touched functions are already
in docs/provenance.md.
Full Changelog: v0.1.0...v0.1.1
v0.1.0
The first public shape of the library. It is a work in progress: the classic
.cbh family is read and exported, .cbv/.cbz archives unarchive in full, and
2CBH yields tags but no moves. README.md § "What is not supported yet" is the
authoritative list of what is missing.
Added
- The consumer façade (
cbvault::bridge).Database::opentakes a base name or
any member file, resolves siblings case-insensitively, validates the mandatory
set (.cbh .cbg .cba .cbp .cbt .cbc .cbs) and reports the generation, the record
count and which optional members were found.Database::headerslists games
from.cbhand the namebases alone and never opens the moves file; asserted
two ways, by renaming.cbgaway after open and for a tag scan. - A sink-based conversion contract.
GameSinkdeclares once per run what it
needs (wants_keys,wants_annotations) and is then handed one entirely borrowed
GameRefper game in ascending game-number order.for_each_gamewalks
sequentially;convert_parallelcuts the id space into 8,192-record chunks and
has one writer stage deliver them in id order, so the sink sees the same
games, in the same order, with the same payloads, at one thread or ten — asserted
over 4,096 fixture games at 1, 2, 3 and 10 threads and batch sizes 1, 7 and
8,192. A sink that only counts games reaches the allocator zero times per
game, asserted with a per-thread counting allocator. - Tag search over the raw set.
Entities::find_playerand friends resolve a
name to an id once;scan/scan_rangethen compare that id per record and
return matches in ascending game order. ChessBase's tournament namebase is not a
valid BST, so a descent that misses falls back to a verified scan of the 10 MB
.cbt;Found::viasays which path answered. - Position search, unindexed.
for_each_position_keyreplays the source and
reports where a Polyglot key occurs, in parallel, withPositionQuery::every
for progress andPositionQuery::cancel_withfor a cancel that takes effect
within one chunk. Measured at 19.9 M plies/s on one thread, 109.6 M/s on ten. - The
cbvaultCLI with four commands:info,verify,pgnand
archive list/archive extract, all with stable--json. The PGN report
always goes to stderr so stdout stays pure PGN. - PGN export, sequential and Rayon-parallel, byte-identical either way.
export_parallel,export_range,export_range_fromandexport_span.
407,350 of 419,385 games match a ChessBase export byte for byte (97.1 %);
§10 ofdocs/format-spec.mdlists the five deliberate deviations. .cbvarchive container and codec, complete. Magic, the 173-byte member
table, the self-describing member count derived from the geometry, per-record
validation against the table's own redundant 32/64-bit copies, and the block
framing. All four block modes are decoded — stored, LZ, Huffman, and
Huffman-then-LZ — so extraction yields 3,871 of 3,871 members and 100 % of
the reference archive's 3.61 GB, including the 512 MB.cbhand the 1.25 GB
.cbj. Listing touches no part of the data pool. Extracting the archive takes
7.1 s on four threads (504 MB/s decoded), against 70.1–72.6 s foruncbvon
the same machine.- Parity with
uncbvmeasured, not asserted. Every member of every corpus —
twic1134.cbv, the 3,871-member reference archive and the three.cbzsamples
— is byte-identical to the reference process's own output, in both
directions (including files the reference wrote that the table does not name).
Reached under a documented two-room clean room; see
docs/research/03-clean-room-audit.md. .cbz, fully. DES in ECB with three key rules depending on the
password's length — as-is at eight bytes, repeat below, fold above —
each verified against the reference's own sample for that rule. The password
check is one eight-byte read, so opening a protected archive never deciphers it.- A 2CBH reader (
cbvault_format::twocbh): fixed 192-byte records, headers,
annotations, and record framing verified over 220,418 records with zero
violations.GameMoves::is_decoded()isfalseby design — the move codec is
not decoded, so nomoves2is fabricated from it. - DES (FIPS 46-3) implemented and checked against the standard's published
test vectors. - The test-only fixture builder (
cbvault-fixtures,publish = false), so
the format round-trips are exercised in CI with no real database.
Changed
- The chess core is
gigachessand onlygigachess. The ported ancestor's
chesscorelayer was replaced wholesale: move generation, legality, FEN/SAN/UCI,
Chess960 and incremental Polyglot Zobrist all come fromgigachess, and the
internal move currency is its 16-bitmoves2. A CI job fails the build if a
second chess implementation appears in the tree. - Moved to
gigachess0.1.9 and let the engine own the null move, so a CBH
pass (0xffffin themoves2stream) is handled byMove::NULLrather than by
cbvault code. - The walk dispatches on what the sink asked for, not per game: no keys means
the fastplay_fastmake, keys means the hash-maintainingplay_hashed,
annotations off means.cbais never opened. Asking for keys costs 4–8 % of a
pass, which is roughly 12× cheaper than the second pass it replaces. - The PGN export was optimised from 171.9 s (64,899 records/s) to
128.1–130.0 s (87,029 records/s) single-threaded, and to 19.7–20.6 s at ten
threads, for byte-identical output — the SAN body inlined into its node, no
per-gameStringorVec, one buffer and onewrite_allper game, tag heads
as string literals, and an ASCII fast path in the name buffer. - The project is named
cbvault, including every crate, path, binary, env var
and document. Format names keep theircbhspelling (cbvault_format::cbh).
Fixed
- The
.cbvkey derivation for a password that is not eight bytes was wrong,
and wrong silently: a short password was zero-padded and a long one truncated,
where the format repeats and folds. Two of the reference's three.cbzsamples
were unopenable as a result. A wrong DES key is still a valid key, so it
deciphers to noise rather than failing — the existing test had asserted the
wrong behaviour. Fixed, and all three rules are now pinned by tests against the
reference's own samples. - A stale-cache bug in the fast make, where a pass turn was validated against
acheckersvalueplay_fasthad left behind. Fixed ingigachessand
adopted here. - SAN over-disambiguation, which was placing a file or rank hint on every
candidate. The writer now emits the minimal qualifier per the SAN standard,
verified equivalent to the per-candidate test over 13,908,447 moves with zero
differences. Gold parity is unchanged. is_legalon a null move was answered by the wrong question, testing the
opponent's king; it now has a dedicated branch upstream.
Known limitations
These are real and are not fixed; each is described in README.md § "What is not
supported yet".
.cbhreading beats the ancestor it was ported from on every path and every
thread count. Both at the same thread count on 11,151,119 records:verify
39.84 s against 46.79 s single-threaded (1.17x), PGN export 94.43 s against
142.44 s single-threaded (1.51x) and 11.23 s against 33.69 s at ten threads
(3.0x). Both spec budgets are met: parallel export at most 25 s at 8+ threads
and under 8 GiB (13.00 s, 3,107 MB), single-threaded at least 65,000
records/s (94.43 s, 1.8x inside it). Memory is the one place the ancestor
leads, and it is mapped pages rather than allocation:cbtoolhas no mmap
dependency and preads into caller buffers, so it holds 13-14 MB where we hold
1,520 MB of whichcat .cbg > /dev/nullshows 1 MB is accounted as the page
cache rather than resident memory.- Memory clarified against the ancestor's, by measurement rather than
inference. The ancestor has nommapdependency andpreads, so its
resident set is its buffers: 13-14 MB against our 1,520 MB. A control
(cat .cbg > /dev/null, reading all 1.25 GB) measures 1 MB, which shows the
pages are file-backed and evictable rather than ours. The mapping still wins on
time - 39.5 s against 46.3 s single-threaded - so this is a trade-off, not a
defect. DbFile::read_exact, and the allocation it removes.Wide::offsetsread
24 bytes per record throughDbFile::read, which allocates aVec— 11
million times on a whole-database walk, the hottest path in the.cbj
reader. It now reads into a stack buffer.cbvault pgnalso reports
peak_writer, the library's own high-water mark for a worker's writer
buffers, which measures 0 MB after all 11.1 M games.- The 2CBH
.2cbgmove codec is not decoded, and a specific list of 2CBH
header fields remains unknown. A 2CBH database yields tags and annotations but
nomoves2. Database::openrefuses a 2CBH set with a typedMissingFile; reading 2CBH
behind the same façade and the same sink is not done.- A game with variations costs one allocation each, in the decoder's per-game
variation stack. Filterhas no "starts from a set-up position" predicate; the bit is in the
move record and a tag search does not open the moves file.- No fuzzing has been done, no crate is published to crates.io, and the
.cbvcontainer layout is verified on two containers and one 1.74 GB archive.
Notes
- No licensed database content is in this repository. The performance figures
inREADME.mdcome from an 11.1 M-record ChessBase set the maintainer holds a
licence for. Every test that needs it is gated on an environment variable and
skips with a visible "this test did not run; it did not pass" message, so CI is
ne...