Skip to content

CLI: reference-source overlap analysis for the TWIC/Ajedrez cut-off (#142) - #144

Merged
jozef2svrcek merged 6 commits into
mainfrom
twic-ajedrez-cutoff-cli
Jul 23, 2026
Merged

CLI: reference-source overlap analysis for the TWIC/Ajedrez cut-off (#142)#144
jozef2svrcek merged 6 commits into
mainfrom
twic-ajedrez-cutoff-cli

Conversation

@jozef2svrcek

@jozef2svrcek jozef2svrcek commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Adds read-only CLI tooling to measure how much two reference sources duplicate each other, so an installed deep historical base (Ajedrez) can inform the game-date window for the weekly feed (TWIC). Instrument for #142. Single commit on top of main.

What's added

Command Purpose
sources overlap --a <col> --b <col> [--by month|year|none] [--json] Per-bucket count of A-games that have a B-duplicate — same match rule as games dedup — plus a coverage %. The decisive measurement.
sources items <key> [--limit N] List a source's tracked items (TWIC issues) with publication dates + download/import status; maps a cut-off date → starting issue #.
search games --collection <name> Restrict search/count to a collection (local + daemon proxy + serve handler).
sources sync --skip-dedup --max-position-depth <N> Sync a bulk source without dup-scan / position index — both are pure overhead for overlap measurement, and --skip-dedup keeps cross-source duplicates so overlap can see them.

All new analysis commands are read-only and local (consistent with existing --moves-stats / players dedup, which also require --local).

Design note

The earlier plan (thread duplicate provenance through games dedup --json) was dropped — dedup's JSON stream also feeds the server/GUI job protocol, so mutating it is invasive. A dedicated read-only sources overlap command yields the same coverage numbers cleanly and side-effect-free.

Reproducibility

scripts/twic-ajedrez-cutoff.sh — self-contained, repeatable harness on a throwaway DB: syncs Ajedrez, imports a TWIC issue range (--skip-dedup), computes per-month coverage, and prints the suggested cut-off D* at 90/95/99 % plus the starting TWIC issue. Env-configurable (FROM_ISSUE, TO_ISSUE, THRESHOLDS, REUSE, CLEAN).

Tests

  • search::collection_filter_tests::collection_filter_restricts_count
  • tests::sources_overlap_counts_cross_collection_duplicates
  • cargo build / clippy clean; cargo test -p chess-db green (17 passed). Manually smoke-tested end-to-end on synthetic data (collection counts + overlap prefix-match verified).

🤖 Generated with Claude Code

https://claude.ai/code/session_019KdPPg7P3bZ5p2RRhZZcv2

@jozef2svrcek @claude
CLI: reference-source overlap analysis for the TWIC/Ajedrez cut-off (#…
d35332f 
…142)

Adds read-only tooling to measure how much two reference sources duplicate
each other, so an installed deep historical base (Ajedrez) can inform the
game-date window for the weekly feed (TWIC).

- `sources overlap --a <col> --b <col> [--by month|year|none] [--json]`:
  per-bucket count of A-games that have a B-duplicate (same match rule as
  `games dedup`) plus a coverage %.
- `sources items <key> [--limit N]`: list a source's tracked items with
  publication dates + download/import status (maps a cut-off date -> issue #).
- `search games --collection <name>`: collection filter (local + daemon proxy
  + serve handler).
- `sources sync --skip-dedup --max-position-depth <N>`: sync a bulk source
  without dup-scan / position index (overhead for overlap measurement; also
  keeps cross-source duplicates so `overlap` can see them).
- `scripts/twic-ajedrez-cutoff.sh`: self-contained, repeatable experiment
  harness (throwaway DB) that syncs Ajedrez, imports a TWIC range, and reports
  the suggested cut-off date D* at 90/95/99% coverage.

Tests: unit tests for the collection filter and the overlap match rule.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KdPPg7P3bZ5p2RRhZZcv2
@jozef2svrcek jozef2svrcek mentioned this pull request Jul 23, 2026
Closed
3 tasks
jozef2svrcek and others added 4 commits July 23, 2026 18:25
@jozef2svrcek @claude
harness: cache downloaded archives on a persistent path across runs
3cadf53 
`chess-db` already reuses any source archive present on disk (the download
step skips a file that exists, independent of the DB ledger), so the ~740 MB
Ajedrez base and the TWIC zips only need fetching once per machine. Make the
harness exploit that:

- DATA now defaults to a durable ${XDG_CACHE_HOME:-$HOME/.cache}/lpdo-experiments
  (not /tmp, which is wiped on reboot); cached archives live under
  $DATA/ajedrez-otb and $DATA/twic and are reused on every subsequent run.
- FRESH_DB=1 rebuilds the database from the cached archives (no re-download).
- CLEAN=1 now explicitly wipes everything including the cache.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KdPPg7P3bZ5p2RRhZZcv2
@jozef2svrcek @claude
CLI: add sources fide-coverage diagnostic (#142)
6b55a31 
Reports what fraction of games have both / one / neither player carrying a
FIDE ID (game-level, optionally bucketed by year), plus distinct-player
coverage, scoped to a collection or the whole DB.

FIDE ID is the reliable cross-source join key: `games dedup` / `sources
overlap` match on player-row ids, so a source that lacks FIDE IDs (e.g.
Ajedrez) whose player rows don't unify with a source that has them (e.g. TWIC,
which also shortens names "Svrcek, Jozef" -> "Svrcek, J") produces hidden
duplicates and undercounts overlap. This command quantifies that gap so the
suitability of a deep-history source can be judged before measuring overlap.

Read-only, local-only. Unit test covers the game/player classification.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KdPPg7P3bZ5p2RRhZZcv2
@jozef2svrcek @claude
harness: use --fast bulk inserts by default (throwaway DB; FAST=0 to …
791048e 
…disable)

The experiment DB is disposable, so --fast's only downside (not crash-safe)
doesn't apply. Appender-based inserts are markedly faster for the large
Ajedrez load. Add a FAST knob (default 1) covering both the Ajedrez sync and
the TWIC import.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KdPPg7P3bZ5p2RRhZZcv2
@jozef2svrcek @claude
sources items: show per-item imported-game count and game-date span
d885169 
LEFT JOIN games on issue_id to report, per tracked item, how many games it
contributed and their MIN/MAX date. Reveals what date range each file covers
(e.g. the two Ajedrez archives overlap rather than splitting cleanly by date)
and helps map a cut-off date to a starting issue. Also documents the earlier
fide-coverage command in the changelog.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019KdPPg7P3bZ5p2RRhZZcv2
This was referenced Jul 23, 2026
Closed
Open
Open
@jozef2svrcek
Merge branch 'main' into pr144-merge
65f0efb 
# Conflicts:
#	chess-db/src/jobs.rs
@jozef2svrcek
jozef2svrcek merged commit fa9f462 into main Jul 23, 2026
6 checks passed
@jozef2svrcek
jozef2svrcek deleted the twic-ajedrez-cutoff-cli branch July 23, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@jozef2svrcek