feat: Add evolving skill for tasks in code mode

[ncrispino]

feat: Evolving Skills and Previous Session Skill Discovery

Closes MAS-150

Description

This PR introduces Evolving Skills as the central planning mechanism for code-based workflows, along with the ability to discover and reuse skills from previous MassGen sessions.

Key Features

1. Evolving Skills (evolving-skill-creator): A built-in skill that guides agents to create structured workflow plans that:

   • Document detailed workflow steps before execution
   • Inventory available resources (MCP servers, custom tools, other skills)
   • Create reusable Python scripts in tasks/evolving_skill/scripts/
   • Capture learnings for future improvement

2. Previous Session Skill Discovery: New config option load_previous_session_skills: true that:

   • Scans .massgen/massgen_logs/ for evolving skills from past sessions
   • Discovers SKILL.md files in final/agent_X/workspace/tasks/evolving_skill/
   • Makes them available in the system prompt with <location>previous_session</location>
   • Copies skill files to Docker containers for agent access

3. Enhanced SKILL.md Format: Updated evolving-skill-creator to emphasize:

   • Required YAML frontmatter (name and description)
   • Naming guidelines (descriptive like artist-website-builder, not instance-specific)
   • Structured sections: Workflow, Tools to Create, Tools to Use, Skills, Packages, Learnings

Type of change

• New feature (feat:) - Non-breaking change which adds functionality

Files Changed

New Files

• massgen/skills/evolving-skill-creator/SKILL.md - Built-in skill for creating evolving skills
• massgen/configs/skills/skills_with_previous_sessions.yaml - Example config

Modified Files

• massgen/cli.py - Parse load_previous_session_skills from YAML config (4 locations)
• massgen/frontend/web/server.py - Parse config for web frontend (2 locations)
• massgen/orchestrator.py - Pass config to filesystem manager
• massgen/filesystem_manager/_filesystem_manager.py - Pass to Docker manager
• massgen/filesystem_manager/_docker_manager.py - Copy previous session skill files to Docker
• massgen/filesystem_manager/skills_manager.py - scan_previous_session_skills() function
• massgen/system_message_builder.py - Fixed logging (loguru instead of standard logging), scan logs_dir
• massgen/agent_config.py - Added load_previous_session_skills to CoordinationConfig
• docs/source/user_guide/tools/skills.rst - Documentation for evolving skills and previous session discovery

Checklist

• I have run pre-commit on my changed files and all checks pass
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

Pre-commit status

# TODO: Run pre-commit before PR submission
uv run pre-commit run --all-files

How to Test

Prerequisites

• Have at least one previous MassGen session with an evolving skill created
• Docker running (for Docker mode testing)

Test CLI Commands

1. Run with previous session skills enabled:

uv run massgen --config massgen/configs/skills/skills_with_previous_sessions.yaml "Create a website about Bob Dylan"

2. Verify skills are discovered in logs:

# Check the log file for these lines:
grep "load_previous_session_skills" .massgen/massgen_logs/log_*/turn_1/attempt_1/massgen.log
# Expected: [SystemMessageBuilder] load_previous_session_skills = True

grep "previous_session" .massgen/massgen_logs/log_*/turn_1/attempt_1/massgen.log
# Expected: [SystemMessageBuilder] Scanned skills: X builtin, Y project, Z previous_session

3. Test skill discovery directly (Python):

from pathlib import Path
from massgen.filesystem_manager.skills_manager import scan_skills, scan_previous_session_skills

# Test previous session skill scanning
logs_dir = Path(".massgen/massgen_logs")
prev_skills = scan_previous_session_skills(logs_dir)
print(f"Found {len(prev_skills)} previous session skills:")
for skill in prev_skills:
    print(f"  - {skill['name']}: {skill.get('source_path', 'N/A')}")

# Test full skill scanning
skills_dir = Path(".agent/skills")
all_skills = scan_skills(skills_dir, logs_dir=logs_dir)
by_location = {}
for s in all_skills:
    loc = s.get('location', 'unknown')
    by_location[loc] = by_location.get(loc, 0) + 1
print(f"Skills by location: {by_location}")

Architecture

Skill Discovery Flow

1. Config: load_previous_session_skills: true
                    ↓
2. cli.py parses into CoordinationConfig
                    ↓
3. orchestrator.py passes to setup_orchestration_paths()
                    ↓
4. _filesystem_manager.py passes to create_container()
                    ↓
5. _docker_manager.py copies skill files to temp_skills_dir
                    ↓
6. system_message_builder.py scans logs_dir for system prompt
                    ↓
7. Skills appear in agent system prompt with <location>previous_session</location>

Path Structure Scanned

.massgen/massgen_logs/
  └── log_YYYYMMDD_HHMMSS/
      └── turn_N/
          └── attempt_N/
              └── final/
                  └── agent_X/
                      └── workspace/
                          └── tasks/
                              └── evolving_skill/
                                  └── SKILL.md  ← Discovered

Self-Improvement Through Evolving Skills

This feature enables a self-improvement loop where agents learn from past sessions:

Session 1: Agent creates evolving skill for "artist website"
    → Documents workflow, creates scripts, captures learnings
    → Skill saved: "artist-website-builder"

Session 2: Agent gets new task "create a musician portfolio site"
    → Discovers "artist-website-builder" from previous session
    → Reuses scripts (fetch_artist_data.py, build_site.py)
    → Builds on learnings ("use Discogs API instead of AllMusic")
    → Updates skill with new learnings

Session 3+: Each iteration improves the skill
    → Better workflows
    → More robust scripts
    → Richer learnings

Key benefits:

• Knowledge Accumulation: Learnings persist across sessions instead of being lost
• Tool Reuse: Python scripts created in one session are available in future sessions
• Workflow Refinement: The "What Worked / What Didn't" sections guide future execution
• Cross-Agent Learning: Skills from agent_a can be discovered and used by agent_b in later sessions
• Reduced Redundancy: Agents don't recreate the same tools/workflows from scratch

This creates a foundation for agents that genuinely improve at tasks over time, building institutional knowledge that compounds with each execution.

Related Documentation

• Skills System: docs/source/user_guide/tools/skills.rst
• Evolving Skill Creator: massgen/skills/evolving-skill-creator/SKILL.md
• Example Config: massgen/configs/skills/skills_with_previous_sessions.yaml