Skip to content

docs: add plan for workflow chaining#552

Draft
andreatgretel wants to merge 4 commits intomainfrom
andreatgretel/docs/workflow-chaining
Draft

docs: add plan for workflow chaining#552
andreatgretel wants to merge 4 commits intomainfrom
andreatgretel/docs/workflow-chaining

Conversation

@andreatgretel
Copy link
Copy Markdown
Contributor

Summary

  • Adds a design plan for workflow chaining - sequencing multiple generation stages where each stage's output seeds the next.
  • Primary use cases: explode (few seeds -> many records), filter-then-enrich, generate-then-judge, multi-turn construction.
  • Secondary benefit: enables removal of allow_resize and simplification of sync/async engine convergence.

What's in the plan

  • Pipeline class in the interface layer with add_stage(), run(), between-stage callbacks
  • to_config_builder() convenience on results for lightweight notebook chaining
  • allow_resize removal (phased, after pipeline ships)
  • Pre-batch processor resize lockdown (fail-fast on row-count changes)
  • Stage-level checkpointing and resume with fingerprint-based invalidation
  • Four concrete use cases with code examples
  • Open questions on image forwarding, fan-out semantics, and downstream seeding scope

Phases

  1. Pipeline class + to_config_builder() (can ship independently)
  2. Remove allow_resize (deprecate first, then remove)
  3. Stage-level resume
  4. Auto-chaining from single config (future)

No code changes - plan document only.

Proposes replacing the in-place allow_resize mechanism with a Pipeline
class that chains multiple generation stages. Each stage gets a fresh
fixed-size tracker, and resize becomes a between-stage concern.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant