Skip to content

Split run_plan_pipeline.py into individual stage files#491

Merged
neoneye merged 12 commits intomainfrom
split-run-plan-pipeline
Apr 2, 2026
Merged

Split run_plan_pipeline.py into individual stage files#491
neoneye merged 12 commits intomainfrom
split-run-plan-pipeline

Conversation

@neoneye
Copy link
Copy Markdown
Member

@neoneye neoneye commented Apr 2, 2026

Summary

  • Split run_plan_pipeline.py (4,257 lines) into ~64 individual stage files under stages/, each containing one Luigi task class
  • Slimmed run_plan_pipeline.py to 562 lines (framework only: PlanTask, ExecutePipeline, utilities)
  • Created stages/full_plan_pipeline.py as the DAG orchestrator importing all stages
  • Updated AGENTS.md with convention for new stages
  • Marked Issue 1 fix step 3 as done in remediation roadmap

This enables:

  • Multiple agents working on different pipeline steps without merge conflicts
  • self_improve/ targeting individual step files
  • Easy DAG insertion (create new file, update downstream requires())

What does NOT change

  • No behavioral changes — identical DAG, identical task execution, identical output files
  • No new dependencies
  • No changes to worker_plan_database/app.py or any external consumer
  • Existing tests pass unchanged

Test plan

  • All stage files pass Python syntax validation
  • Verbatim extraction verified via AST comparison (6 representative files spot-checked)
  • Import chain verified: all public symbols importable from run_plan_pipeline
  • _task_class_to_step_label test still passes
  • CI pipeline tests

🤖 Generated with Claude Code

neoneye and others added 12 commits April 2, 2026 18:23
…plit

Design spec and 11-task implementation plan for splitting the 4,257-line
run_plan_pipeline.py into ~66 individual stage files under stages/.
Optimized for parallel agent work, self_improve/ integration, and
conflict-free DAG insertion.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract StartTimeTask, SetupTask, RedlineGateTask, PremiseAttackTask,
IdentifyPurposeTask, PlanTypeTask into stages/ directory.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ance)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ort)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ipeline from stages

Remove all ~66 task class definitions from run_plan_pipeline.py (3761 lines
deleted) and create stages/full_plan_pipeline.py as the pipeline orchestrator
that imports all task classes from individual stage files. ExecutePipeline.setup()
now uses a deferred import of FullPlanPipeline to avoid circular dependencies.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@neoneye neoneye merged commit ac973af into main Apr 2, 2026
3 checks passed
@neoneye neoneye deleted the split-run-plan-pipeline branch April 2, 2026 18:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant