feat(ai-markmap-agent): ship evolved Markmap generation pipeline (refinement mode, versioning, translation, compression, and docs) #46

lufftw · 2025-12-13T16:56:58Z

Summary

This PR delivers a major end-to-end overhaul of tools/ai-markmap-agent/, evolving it into a scalable, debuggable, and configurable Markmap generation system. It introduces a complete multi-agent workflow, robust data loading + compression, secure runtime API key handling, versioned outputs, translation/translate-only utilities, improved HTML rendering, and refreshed documentation + MkDocs navigation.

Key Changes

1) Evolved pipeline + architecture

Implement a full multi-agent pipeline for Markmap generation, iterating through structured expert feedback and producing finalized outputs.
Introduce a refinement-mode workflow (start from a strong baseline and improve it) with:
- Domain experts (Architect / Professor / Engineer)
- Two-round discussion (independent review → full discussion)
- Code-based consensus via majority voting (2/3)
- Configurable refinement scope and suggestion counts
Add architecture documentation (DESIGN_V2/V3/V4 lineage) and consolidate to the latest workflow.

2) Prompt system redesign (persona/behavior separation)

Convert prompts from .txt → .md
Split prompts into persona and behavior files for maintainability and reuse
Add dynamic prompt selection support (prompt_mode + meta-prompts)
Add role-specific prompt directories (planner/strategist/evaluator/writer/translator) with clear responsibility boundaries

3) Data sources + secure runtime config

Add configurable data_sources in config.yaml (ontology / problems / patterns / roadmaps)
Add config_loader.py with secure runtime API key input (never stored; cleared on exit)
Add a main.py CLI entry point with flags like --dry-run, --no-openai, etc.
Improve config comments/docs (e.g., problems load_mode behavior)

4) Token efficiency: compression + long-content handling

Implement comprehensive input compression to reduce prompt size dramatically:
- Problems / ontology / roadmaps compressed formats
- Short-key mappings (e.g., i/t/d/s/sf) and auto-detection of solution status
Add compressor prompts/behavior for long content handling
Add URL templates config for consistent GitHub/LeetCode linking and deferred link decisions

5) Translation support + translate-only utility

Add language generation strategies: generate vs translate
Add TranslatorAgent and a translation-only script (translate_only.py) to translate existing outputs without rerunning the full pipeline
Fix filename corruption bugs when replacing language suffixes and ensure MD/HTML output directories stay synchronized
Add cleanup logic that preserves YAML frontmatter and removes LLM artifacts
Improve zh-TW translation quality by enforcing Taiwan CS terminology (avoid Mainland wording)

6) HTML output + Markmap template fixes

Fix runtime Markmap errors (e.g., markmapLib is not defined) by aligning correct CDN paths
Upgrade the HTML template to a more consistent UI (toolbar + controls + branding)
Ensure consistent HTML generation paths (e.g., docs/pages/mindmaps/)

7) Debugging + developer ergonomics

Add debug output configuration and phase-by-phase output saving
Save full LLM call inputs/outputs (with model + timestamps + content lengths) to verify prompts and responses
Add data summaries + warnings when sources are empty
Expand .gitignore to exclude generated outputs, debug artifacts, logs, local env files, and ChromaDB persistence

8) Documentation + MkDocs integration

Add/refresh agent design docs and move detailed content behind references from main README
Update MkDocs navigation to include missing pages (patterns, mindmaps, ontology design, local docs build, content guide)

Output Artifacts

Versioned outputs under outputs/versions/vN/ with auto-incrementing runs
Final outputs follow the new naming template: neetcode_ontology_agent_evolved_{lang}.md/html
Translate-only mode can generate synchronized MD + HTML into the configured final directories

Notable Fixes

Resolved “missing prompt variable” warnings across agents by aligning prompt variables with code
Fixed context-length failures by upgrading certain calls to larger-context models when needed
Prevented accidental deletion of old versions on pipeline failure (safe rollback behavior)

Breaking / Behavioral Changes

Pipeline architecture and configs were heavily reorganized; legacy pipelines were removed in favor of the current evolved workflow.
Output naming + directory conventions changed to support version history and consistent publishing paths.

Testing / Validation

Verified pipeline phases run end-to-end with debug outputs enabled
Verified translate-only mode produces synchronized MD + HTML in final directories
Validated filename handling for language suffix replacement and frontmatter preservation

Notes for Reviewers

Most changes are contained within tools/ai-markmap-agent/ plus documentation and MkDocs nav updates.
Recommend reviewing by areas:
1. config + data loading
2. compression + translation
3. pipeline graph + agents
4. writer + HTML converter/template
5. docs + mkdocs navigation

… separation - Rename all prompts from .txt to .md format - Split each agent prompt into persona + behavior files - Create 3 distinct optimizer personas (Structuralist, Semanticist, Pragmatist) - Update config.yaml with gpt-5/5.1/5.2 for optimizers, gpt-5.2 for summarizer, gpt-4 for judges - Add compressor behavior prompt for long content handling - Organize prompts into subdirectories by role type - Add separate section for tools/ai-markmap-agent/ - Ignore output files (intermediate and final) - Ignore ChromaDB persistence data - Ignore logs and local env files

…prompt support - Replace 3 optimizer roles with expert perspectives: - Software Architect (Dr. Alexander Chen) - Algorithm Professor (Prof. David Knuth Jr.) - API Architect (James Patterson) - Add prompt_mode config for static vs dynamic prompt selection - Add meta-prompts for AI-generated personas and behaviors - Add suggest_optimizer_roles.md for AI role suggestions - Convert all prompts to English - Update config.yaml with dynamic_config for each optimizer

…ime API key input - Add data_sources section in config.yaml for configuring ontology, problems, patterns, and roadmaps to read from - Create config_loader.py with runtime API key input via getpass (keys never stored, cleared on program exit) - Create data_sources.py for loading TOML data from configured sources - Create main.py entry point with CLI options (--dry-run, --no-openai, etc.) - Update requirements.txt with tomli for Python < 3.11 compatibility - Update README.md and README_zh-TW.md with new documentation

…tions

- Define languages as list: [en, zh-TW] - Define types with generator mapping: general/specialist - Use template pattern for file naming - Cleaner config for 4 outputs (2 types × 2 languages)

Add all core modules for AI-powered Markmap generation: - agents: BaseAgent, Generator, Optimizer, Summarizer, Judge - memory: ShortTermMemory (STM), LongTermMemory (LTM with ChromaDB) - compression: ContentCompressor for token management - output: MarkMapHTMLConverter for MD→HTML conversion - graph: LangGraph workflow with 5 phases Pipeline generates 4 outputs: - neetcode_general_ai_en.md/html - neetcode_general_ai_zh-TW.md/html - neetcode_specialist_ai_en.md/html - neetcode_specialist_ai_zh-TW.md/html

…guide - Fix prompt variable names to match code: - Optimizers: other_suggestions, previous_feedback, focus_area - Summarizer: suggestions, round_number - Judges: markmap, criteria - Add comprehensive Markmap format guide to all generation prompts - Links, KaTeX math, checkboxes, folding, tables, code blocks - Update config to 1 round for development (production: 3) - Add _format_other_suggestions() to optimizer for debate mode Eliminates all "Missing prompt variable" warnings during execution.

- Add urls config for GitHub/LeetCode link templates - Add data_compression config for token-efficient LLM transmission - Create DataCompressor with 3 formats: compact_json, tabular, minimal - Use short keys (i=id, t=title, d=difficulty, s=has_solution) - Auto-detect solution status from files.solution field - Update prompts with link selection logic: - has_solution=true → GitHub solution URL - has_solution=false → LeetCode problem URL - Replace "LC" with full "LeetCode" in all prompts - Update generator agents to use compressed data format

Add complete data compression for token-efficient LLM transmission: Documentation: - docs/DATA_SOURCES.md: Full specification of input data sources Compression: - Problems: compact_json/tabular/minimal formats (~77% reduction) - Ontology: hierarchy extraction with summaries (~85% reduction) - Roadmaps: learning path compression (~75% reduction) Key mappings: - i=id, t=title, d=difficulty, s=has_solution, sf=solution_file - o=order, rl=role, pq=prerequisite, dt=delta Total token reduction: ~80% (27K → 5.5K tokens for 33 problems)

Add language generation mode config to support two strategies: - "generate": Run full optimization pipeline from scratch (for primary lang) - "translate": Translate from another language's output (fast, for secondary langs) zh-TW now defaults to "translate" mode using gpt-4o, translating from English output instead of running the full multi-round pipeline again. This cuts generation time roughly in half while preserving the option to run independent pipelines per language.

… mode - Fix "markmapLib is not defined" by using correct CDN paths: markmap-lib@0.16.0 and markmap-view@0.16.0 - Implement translate mode for zh-TW language: - Add TranslatorAgent class for LLM-based translation - Modify create_generators to only create for mode="generate" languages - Add create_translators function for translate-mode configs - Update workflow to support translation: - Add Phase 4 (Translating) between Judging and Finalizing - Only mode="generate" languages go through optimization rounds - mode="translate" languages are translated after optimization This reduces generation time by ~50% as zh-TW no longer runs the full multi-round optimization pipeline.

Document new design with key improvements: - Deferred link processing (only in Writer phase) - Configurable judges count and individual models - Multi-round debate mechanism with consensus threshold - New dedicated Markmap Writer for final output - Complete Markmap format guide This reduces prompt burden in early phases and provides better separation of concerns.

…ocessing - Add configurable judges with individual model settings - Add writer config with format_guide path - Add debate settings (consensus threshold, max rounds) - Add post_processing for LC → LeetCode text replacement (code-based) - Update DESIGN_V2.md with Phase 6 post-processing docs

V2 Implementation: - Add WriterAgent for final Markmap generation with links and formatting - Add structured judge feedback (strengths, improvements, consensus) - Implement multi-round debate mechanism between judges - Add PostProcessor for LC → LeetCode text replacement (code-based) - Update graph.py with 7-phase V2 workflow New files: - prompts/writer/writer_persona.md - prompts/writer/writer_behavior.md - prompts/writer/markmap_format_guide.md - src/agents/writer.py - src/post_processing.py Modified: - src/agents/judge.py (debate support) - src/graph.py (V2 workflow) - templates/markmap.html (fix markmapLib error)

…or testing - Add debug_output section to config for saving intermediate outputs - Create src/debug_output.py module for phase-by-phase output saving - Update graph.py to save debug outputs at each phase - Switch all models from gpt-5.x to gpt-4 for cheaper testing - Keep original model names as comments (e.g., # ORIGINAL: gpt-5.2) for easy restoration later

…r agents - Replace non-existent _call_llm() with proper BaseAgent methods: self._build_messages() + self.llm.invoke() - Add missing process() method to TranslatorAgent to satisfy abstract base class requirement - Extract response.content from LangChain response object

- Add llm_calls config section in debug_output - Add _save_llm_call_input() and _save_llm_call_output() to BaseAgent - Save full prompts and responses for all agent types - Support markdown and JSON output formats - Helps debug and verify LLM prompts during development

…M input details - Add detailed LLM input format with timestamp, model info, content length - Add data summary output at Phase 1 (problems, ontology, roadmaps count) - Add warnings when data sources are empty - Helps verify data is correctly loaded and passed to LLM

… format - Update templates/markmap.html with topbar buttons (Fit View, Expand All, Collapse All) - Add markmap-toolbar CDN import for native toolbar support - Include NeetCode branding in toolbar - Update fallback template in html_converter.py to match - Use consistent styling with border and hover effects

- Correct base_paths in config.yaml (3 levels up from config/ dir) - Escape {sf} and {slug} in behavior prompts to prevent format errors

Generalist and Specialist agents were using gpt-4 (8K context) but input prompts exceeded the limit. Changed to gpt-4o (128K context).

…rkflow - Add planners/ directory with generalist and specialist planner prompts - Add strategists/ directory with architect, professor, and UX strategist prompts - Add integrator/ directory with integration and conflict resolution prompts - Add writer_behavior_v3.md for Structure Spec to Markdown conversion - All prompts follow V3 design: discuss concepts, not formatting - Strategists output YAML suggestions, only Writer produces Markdown

V3 core changes: - Add Structure Specification schema (YAML-based intermediate format) - Replace Generators with Structure Planners (output YAML, not Markdown) - Replace Optimizers with Content Strategists (discuss concepts, not formatting) - Add Integrator with consensus detection (replaces Summarizer) - Replace Judges with Evaluators (assess Structure Spec quality) - Update Writer to render Structure Spec → final Markdown New files: - src/schema/structure_spec.py - Complete spec schema & validation - src/agents/planner.py - Generalist/Specialist planners - src/agents/strategist.py - Architect/Professor/UX strategists - src/agents/integrator.py - Suggestion integration with consensus - src/agents/evaluator.py - Structure/Content evaluators - src/agents/writer_v3.py - Final Markdown renderer - src/graph_v3.py - V3 LangGraph pipeline - prompts/evaluators/*.md - Evaluator behavior prompts Benefits: - ~80% token reduction in multi-round discussions - Clear separation of process (YAML) and product (Markdown) - Scalable N-strategist architecture with dynamic consensus Usage: python main.py (V3 default), python main.py --v2 (legacy)

The planner agents were incorrectly loading generator prompts which expect {metadata}, but planners pass {problems}. Added planner-specific model configs (generalist_planner, specialist_planner) pointing to the correct planner behavior/persona prompts. Fixes "Missing prompt variable: 'metadata'" warning.

…ssues - Add content_strategist, evaluator, integrator configs to avoid fallback to legacy V2 configs (optimizer, judges, summarizer) - Add TECHNIQUE_VARIANT to GroupingType enum for specialist planner - Change writer model from gpt-4 (8K) to gpt-4o (128K context) - Fix integrator variable names to match prompt expectations - Update judge prompts to use structure_spec instead of markmap Fixes warnings: - Missing prompt variable: 'current_markmap' - Missing prompt variable: 'markmap' - Prompt file not found: judge_structure_*.md - Context length exceeded (8192 tokens) - 'technique_variant' is not a valid GroupingType

- Remove V2 pipeline imports (src.graph) - Remove --v2 CLI argument - Simplify banner without version parameter - Remove version switching logic - Streamline workflow output to V3 only

- Delete V2 agents: generator.py, optimizer.py, judge.py, writer.py, summarizer.py - Delete V2 pipeline: src/graph.py - Create translator.py (extracted from generator.py) - Update agents/__init__.py to export V3 agents only - Update graph_v3.py to import from translator.py

Config: - Remove V2 models: generalist, specialist, optimizer, summarizer, judges - Remove V2 workflow settings - Update debug output phase names Files: - Rename writer_v3.py → writer.py (WriterAgentV3 → WriterAgent) - Rename graph_v3.py → graph.py (run_pipeline_v3 → run_pipeline) - Update all imports in main.py, agents/__init__.py

…h error gpt-4 only has 8K context which caused translation failures when the combined input + max_tokens exceeded the limit.

- Refinement mode: improve existing high-quality markmap - 2-round discussion: independent review + full discussion - Code-based consensus: majority vote (2/3), no AI Integrator - Natural language suggestions: leverage AI's strength - Scalable: 2N+1 API calls, fixed 3 sequential batches - Initial implementation: 3 experts (Architect, Professor, Engineer)

- Replace creation mode with refinement mode (start from baseline) - Add domain-specific experts: Architect, Professor, Engineer - Implement programmatic consensus calculation (2/3 majority voting) - Create 2-round discussion workflow (Independent Review + Full Discussion) - Add refinement scope configuration for controlled modifications - Update prompts to English with optimized expert behavior - Increase suggestions to 5-10 per expert (was 3-5) - API efficiency: 2N+1 calls, 3 sequential batches (fixed)

Fix language code replacement in translation phase to only replace the language suffix, not all occurrences of the language code string. This prevents incorrect filenames like "neetcode_gzh-TWeral_ai_zh-TW" when translating from "general_en" to "general_zh-TW". - Use rsplit("_", 1) to parse output_key format correctly - Only replace language code at the end of the key - Improve html_converter.py parsing logic for robustness

- Change filename template to neetcode_ontology_agent_evolved_{lang} - Add outputs/versions/ directory for version control (v1, v2, ...) - Auto-increment version numbers on each run - Update DESIGN_V4.md, README.md, README_zh-TW.md with new naming

- Add versioning.mode config: "continue" | "reset" - Continue mode: load from latest version (vN), produce vN+1 - Reset mode: prompt to delete all versions, start fresh from baseline - Add prompt_on_reset config option - Update DESIGN_V4.md, README.md, README_zh-TW.md with documentation

- Move version deletion from start to end of pipeline - Old versions are only deleted after successful completion - If pipeline fails, old versions are preserved (safe rollback) - Update documentation to reflect safer behavior

Add translate_only.py to translate existing markmap outputs without invoking the full generation pipeline, significantly reducing token consumption.

…suffix - Only replace language suffix at the END of filename - Prevents "general" from becoming "gzh-TWeral" - Add backwards compatibility for old format filenames

- Use correct HTML output dir from config instead of same dir as MD - Add clean_translated_content() to remove LLM artifacts (---, ```, empty lines) - Share cleanup function between translate_only.py and main pipeline

- Fix clean_translated_content() to preserve opening --- for YAML frontmatter - Auto-add --- if content starts with YAML key (e.g., title:) but missing --- - Fix MD file: add missing opening --- - Fix HTML: generate to correct directory (docs/pages/mindmaps/) - Delete wrongly placed HTML from docs/mindmaps/

- Add Taiwan vs Mainland China terminology comparison table - Key differences: 指標(TW) vs 指針(CN), 原地 vs 就地, 列舉 vs 枚舉 - Fix existing zh-TW file with correct Taiwan terms - Make prompt generalizable for future translations

…ories Problem: - MD output was saved to input file's parent directory - HTML output used final_dirs.html from config - This caused MD/HTML files to be out of sync when input came from version history (outputs/versions/v1/) Solution: - Both outputs now use final_dirs from config - MD → final_dirs.markdown (docs/mindmaps/) - HTML → final_dirs.html (docs/pages/mindmaps/) Impact: - translate_only.py --html now produces synchronized outputs - Consistent with main pipeline behavior - Files are always in expected final directories

- Add neetcode_ontology_agent_evolved links (EN/zh-TW) before AI version - Replace detailed architecture with concise feature summary - Reference tools/ai-markmap-agent/README.md for Evolved Agent details - Reference tools/README.md for Basic AI configuration

- Add Two Pointers pattern documentation - Add Agent Evolved mind maps (en/zh-TW) - Add Ontology Design reference - Add Local Docs Build Options guide - Add MkDocs Content Guide

lufftw

I’ve reviewed the changes in this PR, and everything looks good. The logic is correct, and the code is clean and consistent with the rest of the project. Approving.

lufftw added 30 commits December 13, 2025 13:40

docs(config): add detailed English comments for problems load_mode op…

66eba5e

…tions

refactor(ai-markmap-agent): simplify output naming config structure

dd14907

- Define languages as list: [en, zh-TW] - Define types with generator mapping: general/specialist - Use template pattern for file naming - Cleaner config for 4 outputs (2 types × 2 languages)

chore: add outputs/debug/ to gitignore for ai-markmap-agent

609ad53

fix(ai-markmap-agent): fix data path resolution and prompt escaping

e7392a8

- Correct base_paths in config.yaml (3 levels up from config/ dir) - Escape {sf} and {slug} in behavior prompts to prevent format errors

fix(ai-markmap): use gpt-4o for generators to fix context length error

d860f71

Generalist and Specialist agents were using gpt-4 (8K context) but input prompts exceeded the limit. Changed to gpt-4o (128K context).

chore(config): upgrade summarizer model to gpt-4o for larger context

ace49d4

refactor(ai-markmap-agent): remove V1/V2 pipeline, keep only V3

c3c459c

- Remove V2 pipeline imports (src.graph) - Remove --v2 CLI argument - Simplify banner without version parameter - Remove version switching logic - Streamline workflow output to V3 only

lufftw added 17 commits December 13, 2025 19:07

fix(ai-markmap-agent): use gpt-4o for translator to fix context lengt…

ccf195a

…h error gpt-4 only has 8K context which caused translation failures when the combined input + max_tokens exceeded the limit.

feat: add translation-only script to avoid full pipeline execution

d0153fe

Add translate_only.py to translate existing markmap outputs without invoking the full generation pipeline, significantly reducing token consumption.

fix(translate_only): fix filename corruption when replacing language …

a6ff414

…suffix - Only replace language suffix at the END of filename - Prevents "general" from becoming "gzh-TWeral" - Add backwards compatibility for old format filenames

docs(mkdocs): add 6 missing files to navigation

d4000e7

- Add Two Pointers pattern documentation - Add Agent Evolved mind maps (en/zh-TW) - Add Ontology Design reference - Add Local Docs Build Options guide - Add MkDocs Content Guide

lufftw commented Dec 13, 2025

View reviewed changes

lufftw merged commit 1beb0f6 into main Dec 13, 2025

lufftw deleted the feat/multi-agent-configurable-markmap-pipeline branch December 13, 2025 16:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(ai-markmap-agent): ship evolved Markmap generation pipeline (refinement mode, versioning, translation, compression, and docs) #46

feat(ai-markmap-agent): ship evolved Markmap generation pipeline (refinement mode, versioning, translation, compression, and docs) #46

Uh oh!

lufftw commented Dec 13, 2025

Uh oh!

lufftw left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat(ai-markmap-agent): ship evolved Markmap generation pipeline (refinement mode, versioning, translation, compression, and docs) #46

feat(ai-markmap-agent): ship evolved Markmap generation pipeline (refinement mode, versioning, translation, compression, and docs) #46

Uh oh!

Conversation

lufftw commented Dec 13, 2025

Summary

Key Changes

1) Evolved pipeline + architecture

2) Prompt system redesign (persona/behavior separation)

3) Data sources + secure runtime config

4) Token efficiency: compression + long-content handling

5) Translation support + translate-only utility

6) HTML output + Markmap template fixes

7) Debugging + developer ergonomics

8) Documentation + MkDocs integration

Output Artifacts

Notable Fixes

Breaking / Behavioral Changes

Testing / Validation

Notes for Reviewers

Uh oh!

lufftw left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants