Taxonomy gen

[drewsungg]

Summary

Reworks the taxonomy-gen subproblem pipeline to produce usable GRPO training data and commits the first curated dataset for conics-tangent-5.

The pre-existing pipeline hit three dead ends in practice:

1. Template lock-in — the generator kept rewriting the same problem with only the numerical parameter changed, so the agreement-band filter rejected 100% of attempts on easy skills (e.g. "dimension of degree-$d$ plane curves" always landed at agreement=1.00 → too_easy).
2. Long-run auth expiry — the in-memory Vertex ADC token baked into the OpenAI client would expire mid-run, forcing a kill + manual re-auth every ~hour and losing skill-level progress.
3. No dedupe — even when keeps accumulated, the filter couldn't catch near-duplicate subproblems that would give bad GRPO signal.

This PR addresses all three and ships a first usable dataset.

What changed

Generator diversity (Stage1/taxonomy_generation.py)

• Memory trick (§3.2 of arxiv 2409.04109). Each generate_for_skill call now threads the last 50 prior attempts (truncated problem text + final answer) back into GENERATE_PROMPT. The generator is explicitly told to produce non-paraphrases with different numerical answers.
• Structural-diversity rule. New load-bearing clause in GENERATE_PROMPT: a problem that reads as a find-and-replace of a prior attempt (same template, different number) is a failure. The prompt enumerates concrete axes of variation (object, algebraic incarnation, question direction, ambient setting).
• Parametric-family rule in DECOMPOSE_PROMPT. Rule 6b rejects skill descriptions pinned to a single instance — every description must name a free parameter ($d$, $n$, $k$) or point to a technique that naturally applies to many configurations. Self-audit step (f2) checks "could 10+ non-identical subproblems exist for this skill?".

Real-time observability

• on_keep / on_skip callbacks flush keeps.json and skips.json atomically after every accepted/rejected candidate, not just at end-of-skill. You can jq .n_problems any time during the run.
• per_skill_stats.json refreshes on the same cadence.

Resume + auth resilience

• --start-from-skill N CLI flag skips the first $N{-}1$ skills — useful after crashes or auth expiries. The other_skill_names isolation list still uses the full taxonomy; only generation is skipped.
• Vertex ADC auto-refresh. Both LLM call paths (_robust_completion in taxonomy_generation, call_llm in distinct_llm_prompting) now detect 401/AuthenticationError, call _get_vertex_access_token() to re-read ADC from disk, and mutate client.api_key in place before the next tenacity retry. A shared _is_auth_error(exc) helper classifies by exception-type name + message substring.

Dedupe (Stage1/dedupe_subproblems.py, new)

• BGE-large-en-v1.5 (2023, 335M params, 1024-dim) — chosen over the paper's MiniLM-L6-v2 because math-dense text with high surface-vocabulary overlap smears under smaller embedders.
• Cosine similarity, greedy first-seen, configurable threshold.
• Two modes: --per-skill (dedup inside each skill only; right default for skill-isolated training data) or global (default).
• Atomic writes, preserves all fields of the input keeps payload.

First dataset: datasets/conics-tangent-5/keeps_deduped_095.json

86 subproblems across 10 skills, per-skill dedupe at cosine threshold 0.95.

Skill	Kept (of 15)	Notes
Dimension of plane curve linear system	5	Solver error floor — only borderline arithmetic slips land in-band
Degree of discriminant hypersurface	9
Divisor class of tangency	10
Naive Bézout count	1	Solver error floor — multiplication is mastered content
Fulton excess-intersection correction	13	Strongest signal; clean isolation
Chasles' characteristic numbers	10	Watch for target overlap on $c \ge 1$ problems
Veronese embedding → hyperplane	12	⚠️ see known issues
Lang–Weil point-count estimate	9	⚠️ see known issues
Chebotarev density	6
Degree of étale map (punchline)	11

Threshold tuning. At 0.88 (paper default, stronger embedder): 23/123 survived — too aggressive; BGE on math-dense text collapses genuinely distinct problems that share LaTeX + domain boilerplate. At 0.92: 47/123. At 0.95: 86/123. 0.95 only catches near-identical rewrites (same problem with a different number), which is the right semantic for training data.

Known issues

Both warrant a manual pass before training:

• Skill 7 (Veronese), sample 1. Conflates "$Q$ passes through a point $P$" (legitimately linear under the Veronese embedding) with "$Q$ is tangent to a conic $C_i$" (actually a degree-6 divisor, class $6H$). The generated problem asks for "conics tangent to 5 smooth conics" via the Veronese translation and gets answer 1 at agreement=0.8 — but the correct answer is Chasles' 3264. The solver majority-voted on the false premise; the filter can't catch this class of error.
• Skill 8 (Lang–Weil), off-by-one variant. The "smallest $M$ in $|N(p) - p| \le M\sqrt{p}$ for smooth degree-7 curve" problem lands at answer 31 with agreement=0.9, but the Weil bound is $M = 2g = 30$ for $g = 15$. Likely a systematic solver error from including the Lang–Weil error term without projective-vs-affine disambiguation.

Hand-inspect skill 7's 12 keeps and skill 8's 9 keeps before feeding to GRPO. 15–20 minutes of manual review will catch the canaries.

Production notes

• Three separate pipeline runs produced these keeps (skills 1–2 → skills 3 → skills 4–10) because of auth expiries before the auto-refresh patch. The raw per-run keeps files live under the symlinked runs/ dir (not in this repo); the merged + deduped output is committed to datasets/.
• Token auto-refresh survived a full 3h24m run without intervention after the patch.
• _PRIOR_ATTEMPTS_MEMORY = 50 was tuned after observing 12-entry cycling; at 50 entries × ~250 tokens the memory block is ~12K tokens, comfortably within context.

Test plan

• Manually review skill 7 (Veronese) keeps for the tangency-as-linear conflation
• Manually review skill 8 (Lang–Weil) keeps for the off-by-one M bound
• Spot-check 5 random keeps per skill for answer leakage in the problem statement
• Sanity-run GRPO on a 10-problem subset before committing to full 86
• If dataset count needs to be larger, re-run skills 9 and 10 with a different random seed / temperature and re-merge