feat: Improve agent coms

[ncrispino]

Shadow Agent Architecture for Broadcast Responses

Summary

• Implements shadow agent architecture for handling broadcast responses in parallel
• Shadow agents are lightweight clones that respond to ask_others() without interrupting parent agents
• Removes old "interrupt-and-block" broadcast handling code
• Shadow agents now receive full parent context including current turn's streaming content, tool calls, MCP calls, and reasoning

Motivation

The previous broadcast implementation interrupted agents mid-stream to handle ask_others() requests, causing:

1. Deadlocks when agents waited on each other
2. Lost context when agents were interrupted
3. Complex state management with broadcast queues

The shadow agent architecture solves these issues by:

1. True Parallelization: Shadow agents run completely in parallel via asyncio.gather() without blocking parent agents
2. System Prompt Control: Shadow agents use simplified prompts (preserve identity, remove workflow tools like vote/new_answer)
3. Full Context: Shadow agents receive the parent's complete context including work-in-progress

Changes

New Files

• massgen/shadow_agent.py - Shadow agent spawner implementation with:
  • ShadowAgentSpawner class for spawning and managing shadow agents
  • _build_current_turn_context() to capture full streaming context
  • Debug file saving when --debug flag is used

Modified Files

• massgen/_broadcast_channel.py - Added _spawn_shadow_agents() method for parallel shadow spawning
• massgen/chat_agent.py - Added current turn context tracking:
  • _current_turn_content - text content
  • _current_turn_tool_calls - native tool calls
  • _current_turn_reasoning - reasoning/thinking chunks
  • _current_turn_mcp_calls - MCP tool calls with args/results
• massgen/tool/workflow_toolkits/broadcast.py - Simplified; respond_to_broadcast now deprecated
• docs/source/user_guide/advanced/agent_communication.rst - Updated documentation

Removed

• Old interrupt-and-block broadcast queue code from chat_agent.py
• _broadcast_queue, inject_broadcast, _check_broadcast_queue methods
• _handle_broadcast, _handle_broadcast_inline, _handle_broadcast_background methods

How It Works

1. Agent A calls ask_others("What framework should we use?")
2. Broadcast channel spawns shadow agents for all target agents in parallel
3. Each shadow agent:
   • Shares parent's backend (stateless, safe to share)
   • Copies parent's full conversation history
   • Includes parent's current turn context (content, tool calls, MCP calls, reasoning)
   • Uses simplified system prompt (identity preserved, no workflow tools)
   • Generates tool-free text response
4. All responses collected simultaneously
5. Parent agents receive informational injection ("FYI, you were asked X and responded Y")
6. Agent A receives all responses

Debug Mode

Use --debug flag to save shadow agent context:

uv run massgen --config massgen/configs/broadcast/test_broadcast_agents.yaml --debug "..."

Debug files saved to: .massgen/massgen_logs/<session>/shadow_agents/

Contains:

• current_turn_context.formatted - string passed to shadow
• current_turn_context.raw - detailed breakdown (content, tool_calls, reasoning, mcp_calls)
• Full conversation history
• Shadow's response

Test Plan

• Run broadcast agents config and verify shadow agents spawn in parallel
• Verify parent agents continue working uninterrupted
• Check debug output includes full context (content, tool calls, MCP calls)
• Confirm no truncation of context data
• Test that informational messages are injected into parent agents