Experience Report: HITL Gates for Auto-Run Mode

[mellanon]

Context

Ran a 24-hour parallel session with two agents in separate git worktrees, each using a SpecFlow playbook. The workflow has natural review points where human judgment is needed before proceeding. Currently, auto-run skips past these because there's no pause-notify-resume mechanism.

The Problem

In interactive mode, review gates happen naturally (you're watching). In auto-run, they get skipped. The agents completed full cycles without human input:

• Specs were created → agents immediately proceeded to implementation
• Plans were drafted → agents immediately started coding
• No opportunity for "wait, that's not quite right" before significant work was done

Natural Review Points (Skipped in Auto-Run)

• After spec is written → Human should review spec before implementation begins
• After architecture is planned → Human should approve approach
• After implementation → Human should review code before verification
• After tests pass → Human should validate before marking complete

Two Possible Approaches

Option A: Playbook-Level HITL Markers

Playbook documents could declare HITL gates:

## Step 5: Review Specification

<!-- MAESTRO:HITL reason="Spec ready for review" artifact=".maestro/outputs/SPEC.md" -->

- [ ] Human has reviewed and approved the specification

Maestro pauses at the marker, sends notification with reason/artifact, waits for approval.

Option B: Agent-to-Maestro Signaling

The agent (or underlying tool like SpecFlow) could signal Maestro:

maestro signal --pause --reason "Spec ready for review" --artifact "SPEC.md"

Or via file convention that Maestro monitors.

This would allow integration between workflow tools (like SpecFlow) and Maestro's control loop.

Related

• Issue Feature Request: Playbook Exit Condition Detection for Auto Run #231 - Exit condition detection (same session)
• Issue Allow seeing agent output and progress during and after auto-run mode run #201 - Observability during Auto Run

[pedramamini]

Interesting. Question though, why not solely auto execute the docs to the point you need the human in the loop. To break up your auto run docs into phases, where the end of each phase requires human review?

[mellanon]

Great question! I've actually already built split playbooks with explicit phase exits (SpecFlow_1_Specify, SpecFlow_2_Plan, SpecFlow_3_Implement) - so that pattern definitely works.

The gap I'm trying to address is what happens between phases. Split playbooks put the orchestration burden on the human:

• "Which playbook do I run next?"
• "Did the previous one finish?"
• "Where's the artifact I need to review?"
• "How do I pass context to the next phase?"

With integrated HITL gates, the workflow manages itself - runs until a gate, notifies with the artifact ready for review, waits for approval, then resumes from exactly where it stopped. One workflow, one context, human in the driver's seat for decisions but not for orchestration.

Significant compute goes into spec → feature decomposition. Once that investment is made and approved, the system should just execute autonomously until the next decision point. Making users manage separate playbooks inverts the relationship - the human ends up serving the tool instead of the tool serving the human.

Projects like vibe-kanban are building exactly this pattern - review checkpoints integrated into continuous execution. The workflow knows where it is; the human reviews and approves at natural gates.

That said, I understand this adds complexity to Maestro. If the direction is "keep playbooks simple, users manage sequencing," that's a valid design choice - I can work with split playbooks. Just wanted to share the experience report and see if there was appetite for the integrated approach.

[mellanon]

For reference, here's the main playbook I've been using for spec-driven development:

SpecFlow_Development - A monolithic playbook that orchestrates the full workflow:

SELECT_FEATURE → SPECIFY → PLAN → TASKS → IMPLEMENT (loop) → VERIFY → COMPLETE

It uses the SpecFlow Bundle for the underlying spec-driven methodology - gated phases, interview-driven specs, TDD enforcement, and quality evaluations.

The playbook is designed for multi-feature development with SQLite tracking, and supports parallel execution via git worktrees (which is how I ran the 24-hour session with two agents).

[mellanon]

The underlying agent infrastructure is PAI (Personal AI Infrastructure) - a scaffolding layer for Claude Code that provides:

• Skill system - Domain-specific modules that auto-load based on context
• Hook system - TypeScript lifecycle handlers (session start/stop, tool events, notifications)
• Memory system - Persistent session capture, learnings, and cross-session context
• Voice notifications - ElevenLabs TTS for fire-and-forget status updates

SpecFlow Bundle is installed as a PAI skill, which is how the playbook can call commands like specflow specify, specflow plan, etc. The constitutional gates (PAI-PRINCIPLES.md, TDD-EVALS.md) come from PAI's design philosophy.

The combination is: Maestro (orchestration) → PAI (agent infrastructure) → SpecFlow (spec-driven methodology)

[pedramamini]

Small world. I'm talking to Miessler today about PAI, I need to get your very setup going with PAI living under Maestro. Specifically I want it for the agent that runs atop of my Obsidian notebook.

Let me put some more thought into this, it is additional complexity. Let me also propose the following workflow that may assist in the meanwhile. If the documents are ordered correctly and broken into subdirectories for each phase, then the progress is visible in the auto run dropdown menu:

It's less than ideal, but the bottom of the last completed document should provide enough context to press forward. This is the general path I've been taking myself. What you're describing is far cleaner from UX perspective but will need some significant re-working. Let me noodle on this some more.

[mellanon]

Thank you for sharing! I'll check it out. I'm part of Daniel's Unsupervised Learning community and I've had a thread going showing my implementation journey using Maestro, PAI and Specflow Bundle to build an observability extension for PAI.
Mind blowing! I think you are onto something excellent and with a bit more tooling to support the human in the loop around visibility, review and validation it'll be super powerful.

Talking about Obsidian... Check this out. I've been chipping away at building a knowledge domain on the same foundation with PAI on top.

danielmiessler/Personal_AI_Infrastructure#147