feat(apr-cli): #1547 — apr tokenize encode-corpus --estimate-only pre-flight pass

[noahgift]

Summary

GH-1547 piece 3 of 3. Operators dispatched a 47-hour encode run blind in SHIP-TWO-001 5g.1 — a 5-second pre-flight could have shown the projected wall, total tokens, and shard count up front. This PR adds that pre-flight pass.

• New flags: --estimate-only (bool), --estimate-sample-docs <N> (default 1000)
• When --estimate-only is set: NO shards, NO manifest, NO output dir created (short-circuit lives BEFORE create_dir_all)
• Operator-facing stderr block:

  [estimate] input_docs=N
  [estimate] sample_size=K sample_tokens=T sample_wall=Ws
  [estimate] estimated_total_tokens=NNN
  [estimate] estimated_shards=NNN (at shard_tokens=NNN)
  [estimate] estimated_wall=NNN seconds (at --num-workers=N)
  

• AC4 extrapolation formula respects --num-workers:
  estimated_wall = (sample_wall / sample_size) × total_docs / max(num_workers, 1)
• Pure-function extrapolate_estimate kernel — testable without invoking BPE tokenizer or filesystem
• Total-docs counting: JSONL = wc -l style; parquet = sum of row-group metadata footers (no row-group decode)
• Edge cases: 0 sample_size → all-zero; 0 shard_tokens → 0 shards; 0 num_workers → clamp to 1

Tests

5 new unit tests, all passing:

• estimate_only_extrapolation_formula_correct — AC4 math + edge cases (FALSIFY-APR-TOK-PAR-011)
• estimate_only_no_shards_written — AC1 (FALSIFY-APR-TOK-PAR-012)
• estimate_only_no_manifest_written — AC2 (FALSIFY-APR-TOK-PAR-013)
• estimate_only_emits_estimate_lines_to_stderr — wiring smoke (FALSIFY-APR-TOK-PAR-014)
• estimate_only_rejects_zero_sample_size — fail-fast typo guard

Contract

contracts/apr-tokenize-parallel-bpe-v1.yaml v1.2.0 → v1.3.0. Adds estimate_extrapolation equation, four new falsifiers (011/012/013/014), two new proof obligations. pv validate passes.

Sequencing

Branched from PR #1552 (feat/tokenize-encode-corpus-progress-emission) since both PRs touch the same run_encode_corpus signature. Will rebase onto main cleanly once #1552 lands. Auto-merge armed; will queue behind #1552.

Test plan

• cargo build -p apr-cli --features training
• cargo clippy -p apr-cli --features training -- -D warnings
• cargo fmt --package apr-cli -- --check
• cargo test -p apr-cli --features training --lib commands::tokenize::tests (24/24 pass)
• cargo test -p apr-cli --features training --test cli_commands (8/8 pass)
• pv validate contracts/apr-tokenize-parallel-bpe-v1.yaml

🤖 Generated with Claude Code

[noahgift]

Triaged by autonomous rebase pass (2026-05-11): rebase onto main hits a structural conflict that needs manual review.

Specifically, this PR's run_estimate_only_path is typed against &entrenar::tokenizer::BPETokenizer, but main (post #1585 / #1596) has run_encode_corpus wrap the tokenizer in a local EncodeTokenizer enum (Hex / ByteLevel). Three options to unblock:

1. Lift EncodeTokenizer to module scope and retype run_estimate_only_path against &EncodeTokenizer (preferred — single source of truth for the two-format dispatch).
2. Make run_estimate_only_path generic over a closure FnMut(&str) -> Result<Vec<u32>, String> and pass |t| tokenizer.encode(t) at the call site.
3. Take the byte-level branch only when the format is detected; keep the original BPETokenizer arg and skip estimate-only on byte-level.

The conflict also drops one of two test suites (PR's estimate_only_* vs main's repair_manifest_*); whichever resolution path lands needs to keep both. Leaving as-is for the author to pick a direction.