Skip to content

feat: add workflow extraction Pass 2 (transcript to structured workflow)#152

Merged
abrichr merged 1 commit into
mainfrom
feat/workflow-extraction-pass2
Mar 20, 2026
Merged

feat: add workflow extraction Pass 2 (transcript to structured workflow)#152
abrichr merged 1 commit into
mainfrom
feat/workflow-extraction-pass2

Conversation

@abrichr
Copy link
Copy Markdown
Member

@abrichr abrichr commented Mar 20, 2026

Summary

  • Implements Pass 2 of the 4-pass workflow pipeline: extracts structured Workflow objects from EpisodeTranscript (Pass 1 output)
  • Uses a single VLM call per transcript to identify and merge related actions into high-level WorkflowStep entries (e.g., click + type -> "Enter header 'Year' in cell A1")
  • Includes graceful fallback: if VLM parsing fails, maps each transcript entry 1:1 to a WorkflowStep so downstream passes (Pass 3 matching) still work

Details

  • New file: openadapt_evals/workflow/pipeline/extract.py with extract_workflow() public API
  • Updated pipeline/__init__.py to export extract_workflow and fix docstring listing all 4 passes
  • Follows the same VLM client pattern as transcript.py (uses vlm_call + extract_json from openadapt_evals.vlm)
  • VLM prompt asks for structured JSON with think/action/expect per step, action classification, parameters, and source entry provenance
  • Corrective actions are filtered out in fallback mode; the VLM prompt instructs skipping them in normal mode

Test plan

  • Verify ruff check and ruff format pass (confirmed locally)
  • Verify from openadapt_evals.workflow.pipeline import extract_workflow imports cleanly
  • Integration test with a real EpisodeTranscript + VLM API key
  • Verify output Workflow is accepted by Pass 3 (match_workflow_to_canonical)

🤖 Generated with Claude Code

Implements Pass 2 of the workflow pipeline: takes an EpisodeTranscript
(Pass 1 output) and uses a VLM to extract structured Workflow objects
with merged WorkflowStep entries. Includes fallback to 1:1 mapping when
VLM parsing fails.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@abrichr abrichr merged commit 721f940 into main Mar 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant