Six step pipeline

[drewsungg]

Six-Step Taxonomy Pipeline for Subproblem Generation

Problem

The naive pipeline (problem → skills → subproblems → dedupe) fails on problems
with high-confidence wrong attractors — where a model's first pattern-match
leads to a plausible but incorrect interpretation. For example, a problem about
"conics tangent to 5 conics" pattern-matches to the classical count of 3264,
but the actual question (average over F_p-points of a finite étale cover)
requires Chebotarev density and harmonic sums. Decomposing under the wrong
interpretation produces subproblems that test irrelevant skills.

Solution

Insert structural analysis and interpretation enumeration before skill
decomposition, and an adversarial coverage critic after. The full pipeline:

1. Structural feature extraction — analyze the problem statement itself
   (answer type, operations, aggregation, regime, named objects, deviations
   from the simplest phrasing, easily-overlooked details). Forces the model to
   attend to structure before committing to any technique.

2. Interpretation enumeration — produce 3+ distinct readings of the
   problem, including the surface-level one. Each interpretation commits to a
   specific answer type, technique, and justification grounded in structural
   features. An ignored_features field enforces honesty.

3. Skill decomposition — produce a union skill set covering all
   interpretations. Each skill is tagged with which interpretations it serves
   and which structural features it addresses. Skills must be atomic,
   load-bearing prerequisites (not vague topic labels).

4. Adversarial coverage critic — clause-by-clause check that every part of
   the problem statement is covered by some skill. Includes a stripping test
   (remove covered clauses, check what remains) and an inverse stripping test
   (could you reconstruct the problem from just the skills?). Loops back to
   step 3 if gaps are found (capped at 2 revision rounds).

5. Per-skill subproblem generation — generates candidates one skill at a
   time with agreement-window filtering (configurable, default 0.60–0.80).
   Each candidate is solved N times (default 20) to get solver consensus.
   Includes memory/diversity feedback: every prior attempt (kept or skipped)
   is fed back to the generator so it actively diversifies. Top-up rounds
   retry under-producing skills with full prior-attempt memory carried over.
   Top-up can be dedupe-aware (counts deduped keeps, not raw) so post-dedupe
   counts meet the minimum.

6. Embedding-based dedupe — BGE-large cosine similarity (default threshold
   0.95). Supports per-skill mode (dedupe within each skill independently) or
   global mode (across all skills).

Key design decisions

• No answer access required. The pipeline never needs ground truth on the
  parent problem. Validation comes from multiple interpretations, coverage
  checking, and solver agreement filtering.

• All outputs cached. Steps 1-4 produce features.json, interpretations.json,
  skills.json, and critic.json. These are reused on subsequent runs for the
  same target problem, so only step 5 (the expensive part) re-runs.

• Incremental writes. keeps.json and skips.json are flushed after every
  candidate, so a crash mid-run doesn't lose progress.

• Domain-agnostic. None of the prompts mention domain-specific vocabulary.
  They refer only to structural features, interpretations, skills, and
  coverage — concepts that apply to math, physics, biology, or any domain
  with well-defined problems.

• Model hardcoded to openai/gpt-oss-120b-maas for all calls (generation,
  solving, structural analysis). No max_tokens, no client-side timeouts.

Files changed

• Stage1/taxonomy_generation.py — main pipeline implementation (1778 lines)
• Stage1/dedupe_subproblems.py — standalone dedupe module (also callable as
  library from the main pipeline)
• subproblem_generation_pipeline.md — full pipeline spec with all prompts
• docs/specs/2026-04-21-taxonomy-generation-design.md — design spec
• docs/plans/2026-04-21-taxonomy-generation.md — implementation plan
• tests/test_taxonomy_generation.py — unit tests for decomposition, skill
  serialization, and model constant

CLI usage

python -u Stage1/taxonomy_generation.py \
    --problem-path data/target-problems/conics.txt \
    --runs-subdir conics-tangent-5-v2 \
    --n-skills 10 --problems-per-skill 15 --max-candidates-per-skill 200 \
    --n-samples 20 --agree-low 0.60 --agree-high 0.80 \
    --gen-workers 4 --max-workers 32 \
    --min-per-skill 10 --max-topup-rounds 2 \
    --dedupe-threshold 0.95 --dedupe-mode per-skill

Output structure

runs/<subdir>/
  features.json          # Step 1: structural features (cached)
  interpretations.json   # Step 2: 3+ interpretations (cached)
  skills.json            # Step 3: union skill set (cached)
  critic.json            # Step 4: coverage critique (cached)
  stage1_taxonomy/<timestamp>/
    keeps.json           # Accepted subproblems with solver traces
    skips.json           # Rejected candidates with reasons
    per_skill_stats.json # Per-skill generation stats
    keeps_deduped.json   # Final deduplicated output