The Self-Documenting Knowledge Portal & README Spec

[zaebee]

🎯 Feature Design: The Self-Documenting Knowledge Portal & README Spec

1. Context & Goal

A great developer-facing, agent-native tool needs documentation that is as high-fidelity as its code.

We will design a unified Documentation Suite that immediately explains the "Why" and "How" of CGIS. The crown jewel of this suite is our README.md, which is self-documenting: it automatically embeds live, color-coded, compile-time architecture diagrams generated by CGIS itself during CI/CD.

---

2. Documentation Architecture Map

We establish a 3-tier documentation structure inside the /docs directory:

.
├── README.md                  # 🚀 The Entrypoint: Pitch, Quickstart, Live Self-Graph
└── docs/
    ├── architecture/
    │   ├── HOW_IT_WORKS.md    # ⚙️ The Deep-Dive: 3-Pass Compiler Pipelines & SQLite/DuckDB Schemas
    │   └── ONTOLOGY.md        # 🧬 The Schema: Formal L1-L3 mappings (Nodes, Edges, Domains)
    └── how-to/
        ├── AGENT_ONBOARDING.md# 🤖 The AI Guide: Connecting Cursor, Claude, and using MCP
        └── CLI_USAGE.md       # 💻 The Human Guide: cgis ingest, trace, impact, structure, health

---

3. The "Wow-Effect" README.md Spec

The main README.md must immediately hook the reader by showcasing Self-Parsing and Zero-Noise RAG.

Proposed README Outline:

# 🧠 CGIS: Code Graph Intelligence System

> Transforming source code into a deterministic, computable, and zero-noise architectural model for AI Agents.

[![Continuous Integration](https://github.com/user/cgis/actions/workflows/ci.yml/badge.svg)](https://github.com/user/cgis/actions/workflows/ci.yml)
[![Graph Integrity](https://img.shields.io/badge/endpoint?url=https://raw.githubusercontent.com/user/cgis/main/docs/architecture/health_badge.json)](https://shields.io)

---

⚡ The "Wow" Effect: Self-Modeling Engine

CGIS is a self-referential compiler frontend. It is the only graph engine that documents its own source code dynamically.

Every time we commit to main, CGIS runs its parser on src/cgis/, generates a local graph.db, and automatically injects its own execution call-tree directly into this README!

📊 Live System Architecture (Auto-Generated by CGIS)

graph TD
    n_cli["cgis.cli.ingest (cli.py:54)"] -- CALLS --> n_pipe["cgis.pipeline.IngestionPipeline.run (pipeline.py:36)"]
    n_pipe -- CALLS --> n_ext["cgis.extractors.python_extractor.PythonExtractor.parse (python_extractor.py:60)"]
    n_pipe -- CALLS --> n_res["cgis.resolver.engine.ResolverEngine.resolve (engine.py:60)"]
    n_pipe -- CALLS --> n_db["cgis.storage.sqlite_store.SQLiteStore.save_incremental_batch (sqlite_store.py:255)"]

Loading

---

🚀 Quickstart in 30 Seconds

# 1. Install via uv (ultra-fast)
uv pip install cgis

# 2. Ingest your codebase (Generates optimized, cached graph.db)
cgis ingest ./src --output graph.db

# 3. Analyze impact of a critical change (Who calls this?)
cgis impact "my_package.auth.verify_token"

# 4. Check your overall graph health
cgis health --db graph.db

---

4. The Agentic Onboarding Guide (AGENT_ONBOARDING.md)

This document is specifically tailored for LLMs and developers setting up local AI workflows. It explains how to let Claude/Cursor "self-drive" the codebase.

Core Sections:

1. FastMCP Registration: Step-by-step JSON configs for Claude Desktop (claude_desktop_config.json) and Cursor.
2. The "Self-Driving" Prompts: Concrete prompts to trigger autonomous debugging.
   • Example: "Claude, use cgis_analyze_impact on models.py:Node to find out what files we need to update if we add a new metadata field, then write the PR."
3. Why this beats Vector RAG: A simple comparison showing how CGIS cuts token costs by 85% by emitting 1-depth subgraphs instead of dumping whole files.

---

5. The "How It Works" Deep-Dive (HOW_IT_WORKS.md)

For Knowledge Engineers, Architects, and Compiler Enthusiasts, explaining our execution pipeline.

Core Sections:

1. The 3-Pass Architecture:
   • Pass 1: AST Parsing & Structural Edge Emission (CONTAINS/DECLARES via Tree-sitter).
   • Pass 2: Local Namespace Import Mapping (LNM per file).
   • Pass 3: Linkage & Type Propagation (resolving var.method() to absolute FQN).
2. Dual-Store Database Schema: SQLite (OLTP) for traversing, DuckDB (OLAP) for PageRank and metrics.

---

🧪 TDD Acceptance Criteria

• Test 1 (Documentation Existence): Verify that running the documentation build script or linting step checks that all files in the docs/ tree are present and have valid markdown formatting.
• Test 2 (Readme Anchor Verification): Confirm that README.md contains the strict HTML comment blocks <!-- START_CGIS_GRAPH --> and <!-- END_CGIS_GRAPH --> to ensure the autodoc pipeline can inject diagrams.

---

UPDATES

---

🎯 Feature Design: The Self-Documenting Knowledge Portal & README Spec (Issue #62)

1. Context & Goal

A professional developer-facing, agent-native tool must have documentation that is as high-fidelity and strictly validated as its code.

We will establish a Self-Documenting Knowledge Portal. The core architectural goal is to make documentation a dynamic, self-parsing, and self-validating artifact.

Our main README.md and MCP reference guides will be compiled automatically during CI/CD using CGIS's own graph database, and we will enforce a strict 90% docstring coverage guardrail using interrogate.

---

2. Hardened Documentation Map (The Monorepo Tree)

We establish a 3-tier directory layout inside our monoreпо:

.
├── README.md                  # 🚀 Entrypoint: Pitch, Quickstart, Live Self-Graph
├── scripts/
│   ├── inject_readme_graph.py # Inserts compiled Mermaid into README.md
│   └── generate_mcp_ref.py    # Auto-compiles FastMCP tools schema to Markdown
└── docs/
    ├── architecture/
    │   ├── HOW_IT_WORKS.md    # ⚙️ Technical Deep-Dive: 3-Pass Compiler & DB Schemas
    │   └── ONTOLOGY.md        # 🧬 L1-L3 Formal Ontology Mapping Contracts
    └── how-to/
        ├── AGENT_ONBOARDING.md# 🤖 AI Agent Onboarding: FastMCP, prompts, token metrics
        ├── CLI_USAGE.md       # 💻 Human Manual: ingest, trace, impact, structure, health
        └── MCP_REFERENCE.md   # 📑 [AUTO-GENERATED] Formal API schema of all MCP tools

---

3. GitHub Actions Workflows

A. The Verification Pipeline (.github/workflows/ci.yml)

Enforces strict linting, type-checking, docstring coverage, and runs the fast Bun-powered Vite frontend build.

name: Continuous Integration

on:
  push:
    branches: [ main, dev, "feat/**" ]
  pull_request:
    branches: [ main ]

jobs:
  python-ci:
    name: Python Backend Verification
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Setup Astral 'uv'
        uses: astral-sh/setup-uv@v3
        with:
          enable-cache: true
          python-version: "3.12"

      - name: Install Dependencies
        run: uv pip install -e .[dev]

      - name: Run Ruff (Format & Lint Check)
        run: |
          uv run ruff format --check .
          uv run ruff check .

      - name: Run Interrogate (90% Docstring Coverage Guardrail)
        run: |
          # Fails the build if code documentation falls below 90%
          uv run interrogate -v src

      - name: Run Mypy (Strict Type Checking)
        run: uv run mypy src

      - name: Run Pytest (Includes Self-Parsing & Ontology Compliance)
        run: uv run pytest --cov=src --cov-report=term-missing

  react-ci:
    name: React Frontend (Bun-powered)
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Setup Bun
        uses: oven-sh/setup-bun@v2
        with:
          bun-version: latest

      - name: Install Dependencies
        run: cd ui && bun install

      - name: Lint & Format Check
        run: cd ui && bun run lint

      - name: Run Vite Production Build
        run: cd ui && bun run build

---

B. The Self-Documenting Pipeline (.github/workflows/autodoc.yml)

Runs strictly on merges to main. It self-ingests the codebase, generates Mermaid diagrams, compiles the dynamic MCP reference, and commits changes back with [skip ci].

name: Self-Documenting Architecture Sync

on:
  push:
    branches: [ main ]

permissions:
  contents: write # Grants write permissions to commit generated docs back to main

jobs:
  sync-architecture-docs:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.GITHUB_TOKEN }}

      - name: Setup Python via 'uv'
        uses: astral-sh/setup-uv@v3
        with:
          enable-cache: true
          python-version: "3.12"

      - name: Install CGIS
        run: uv pip install -e .

      - name: Step 1: Self-Ingest Core Engine
        run: uv run cgis ingest ./src --output docs/architecture/self_graph.db

      - name: Step 2: Generate Raw Mermaid Diagrams
        run: |
          mkdir -p docs/architecture/diagrams
          # Compile the execution trace of the core IngestionPipeline.run method
          uv run cgis trace "cgis.pipeline.IngestionPipeline.run" --depth 2 -f mermaid > docs/architecture/diagrams/pipeline_flow.mermaid

      - name: Step 3: Run Python Script to Inject Graph into Root README.md
        run: uv run python scripts/inject_readme_graph.py

      - name: Step 4: Run Python Script to Compile MCP Reference Manual
        run: uv run python scripts/generate_mcp_ref.py

      - name: Step 5: Auto-Commit Back to Main [with skip ci]
        run: |
          git config --global user.name "github-actions[bot]"
          git config --global user.email "github-actions[bot]@users.noreply.github.com"
          
          git add README.md docs/
          
          if ! git diff-index --quiet HEAD; then
            # [skip ci] prevents infinite recursive GHA workflow loops
            git commit -m "docs: auto-sync README.md and MCP reference [skip ci]"
            git push origin main
          fi

---

4. Algorithmic Blueprints (Automation Scripts)

Script 1: README.md Graph Injector (scripts/inject_readme_graph.py)

Performs a safe, idempotent string-block replacement using HTML comment delimiters.

# src/cgis/scripts/inject_readme_graph.py
from pathlib import Path

def inject_graph():
    readme_path = Path("README.md")
    graph_path = Path("docs/architecture/diagrams/pipeline_flow.mermaid")
    
    if not readme_path.exists() or not graph_path.exists():
        raise FileNotFoundError("Missing README.md or pipeline_flow.mermaid file.")

    mermaid_code = graph_path.read_text(encoding="utf-8").strip()
    replacement = f"<!-- START_CGIS_GRAPH -->\n```mermaid\n{mermaid_code}\n```\n<!-- END_CGIS_GRAPH -->"
    
    readme_content = readme_path.read_text(encoding="utf-8")
    start_tag, end_tag = "<!-- START_CGIS_GRAPH -->", "<!-- END_CGIS_GRAPH -->"
    
    if start_tag in readme_content and end_tag in readme_content:
        parts = readme_content.split(start_tag)
        head = parts[0]
        tail = parts[1].split(end_tag)[1]
        
        readme_path.write_text(f"{head}{replacement}{tail}", encoding="utf-8")
        print("✅ Successfully injected latest architecture graph into README.md")

if __name__ == "__main__":
    inject_graph()

Script 2: FastMCP Documenter (scripts/generate_mcp_ref.py)

Performs programmatical introspection of the FastMCP server instance to generate MCP_REFERENCE.md [INDEX_1.1.2, INDEX_1.1.8, INDEX_3].

# src/cgis/scripts/generate_mcp_ref.py
from pathlib import Path
from cgis.api.mcp_server import mcp # Import our active server instance

def generate_reference():
    output_path = Path("docs/how-to/MCP_REFERENCE.md")
    output_path.parent.mkdir(parents=True, exist_ok=True)
    
    lines = [
        "# 📑 MCP Tools Reference Manual",
        "*Auto-compiled from FastMCP docstrings and type annotations.*",
        ""
    ]
    
    # Introspect the registered FastMCP tools
    # E.g. traversing mcp.tools dictionary
    for tool_name, tool in sorted(mcp.tools.items()):
        lines.append(f"## `cgis_{tool_name}`")
        lines.append(f"**Description:** {tool.description or 'No description provided.'}\n")
        
        lines.append("### Arguments:")
        # Generate markdown table for arguments schemas
        lines.append("| Argument | Type | Required | Description |")
        lines.append("| :--- | :--- | :--- | :--- |")
        
        # FastMCP stores arguments in pydantic fields or properties
        for param_name, param in tool.parameters.model_fields.items():
            req_str = "Yes" if param.is_required() else "No"
            lines.append(f"| `{param_name}` | `{param.annotation.__name__}` | {req_str} | {param.description or 'N/A'} |")
            
        lines.append("\n---\n")

    output_path.write_text("\n".join(lines), encoding="utf-8")
    print("✅ Successfully generated docs/how-to/MCP_REFERENCE.md")

if __name__ == "__main__":
    generate_reference()

---

5. TDD Acceptance Criteria

• Test 1 (Docstring Coverage Guardrail): Adding a public function inside src/cgis without a docstring must cause interrogate to exit with code 1 and fail the CI pipeline.
• Test 2 (Idempotent Markdown Replacement): Running inject_readme_graph.py multiple times consecutively must result in the exact same README.md content without duplicating tags.
• Test 3 (FastMCP Schema Introspection): Running generate_mcp_ref.py must successfully extract all four tools (cgis_ingest, cgis_trace_flow, cgis_analyze_impact, cgis_get_structure) and correctly map their Pydantic property schemas.

[zaebee]

The Self-Documenting Knowledge Portal & README Spec

Feature Design (FD) — Final

1. Objective

Create a high-fidelity, agent-native, self-documenting knowledge portal for CodeGraph Brain where documentation is as deterministic, accurate, and up-to-date as the code itself.

The system will automatically maintain:

• A rich, always-current README.md
• Architecture diagrams
• MCP tool reference
• Project health status

2. Documentation Architecture

.
├── README.md                 # 🚀 Main entrypoint: Pitch, Quickstart, Live diagrams & health
├── scripts/                  # 🛠️ Automation Engine
│   ├── inject_readme_graph.py    # Injects Mermaid diagrams into README.md
│   ├── generate_mcp_ref.py       # Auto-generates MCP tools schema to Markdown
│   └── generate_health.py        # Generates health_badge.json for status badges
└── docs/
    ├── architecture/
    │   ├── HOW_IT_WORKS.md       # ⚙️ Technical Deep-Dive (3-Pass Pipeline, DB schemas, etc.)
    │   ├── ONTOLOGY.md           # 🧬 L1-L3 Formal Ontology Mapping
    │   └── diagrams/             # 🖼️ Auto-generated Mermaid (.mmd) files
    └── how-to/
        ├── AGENT_ONBOARDING.md   # 🤖 AI Agent Onboarding (MCP, prompts, metrics, best practices)
        ├── CLI_USAGE.md          # 💻 Human CLI reference (ingest, trace, impact, etc.)
        └── MCP_REFERENCE.md      # 📑 Auto-generated formal API schema of all MCP tools

3. CI/CD & Automation

A. Verification Pipeline (.github/workflows/ci.yml)

• Python: ruff (format + lint), interrogate (≥90% docstring coverage), mypy (strict), pytest + coverage
• React (Bun): bun run lint, bun run type-check, bun run build
• scripts/ directory is included in global linting and type-checking

B. Self-Documenting Pipeline (.github/workflows/autodoc.yml)

Triggered on successful merge to main

1. Self-Ingest
   uv run cgis ingest ./src --output docs/architecture/self_graph.db

2. Diagram Generation
   uv run cgis trace "cgis.pipeline.IngestionPipeline.run" --depth 2 -f mermaid > docs/architecture/diagrams/pipeline_flow.mermaid
   (and other key flows)

3. Health Check
   uv run python scripts/generate_health.py

4. README Injection
   uv run python scripts/inject_readme_graph.py

5. MCP Reference Sync
   uv run python scripts/generate_mcp_ref.py

6. Auto-commit (if changes detected)
   Commit message includes [skip ci]

4. Key Conventions

• All public modules, classes, and functions must have high-quality docstrings.
• interrogate is enforced in CI (minimum 90% coverage).
• Documentation generation must be idempotent.
• Changes in core architecture should be reflected automatically via CGIS tracing.

5. TDD Acceptance Criteria

• New public function without docstring fails CI (interrogate)
• inject_readme_graph.py is idempotent (no duplicate Mermaid blocks on re-run)
• MCP_REFERENCE.md accurately reflects all tools from src/cgis/api/mcp_server.py
• Health badge / health_badge.json correctly updates based on self-graph integrity and other metrics
• README.md contains live, correctly rendered Mermaid diagrams
• Full autodoc.yml workflow succeeds and can commit changes
• AGENT_ONBOARDING.md allows a new AI agent to become productive quickly
• All scripts in /scripts/ pass linting, type-checking and are covered by tests

6. Out of Scope

• Full OpenAPI/Swagger generation
• Multilingual support
• Long-term versioned documentation archive

7. Next Steps After Merge

1. Create folder structure + skeleton files
2. Add interrogate and other dependencies
3. Implement the three scripts in /scripts/
4. Create both CI workflows
5. Update root README.md with injection markers
6. Run first full autodoc cycle

---

Status: Ready for implementation
Priority: Must-have
Owner: @zaebee + AI Teammates

[zaebee]

📝 Review of Feature Design: The Self-Documenting Knowledge Portal & README Spec

The proposed design is outstanding. It moves documentation from a static afterthought to a dynamic, agent-native component of the system. The "Self-Parsing" and "Self-Validating" aspects perfectly embody the core value of CodeGraph Brain.

✅ Strengths

• Automation-First: Automating the sync via GitHub Actions ensures high fidelity with minimal manual effort.
• Agent-Centric: The auto-generated MCP_REFERENCE.md is a critical "onboarding" asset for LLM agents.
• Self-Referentiality: Having the tool parse its own structure is a powerful demonstration of its capabilities.

🛠 Technical Refinements

1. Dependency Management: We must add interrogate to the dev dependency group in pyproject.toml to enforce the 90% docstring coverage guardrail.
2. Async Introspection: Since mcp.list_tools() is an asynchronous method, the generate_mcp_ref.py script should use asyncio.run(mcp.list_tools()) to remain idiomatic and robust.
3. CI/CD Integration: The autodoc.yml workflow is well-designed. Ensure it uses the contents: write permission (as noted) and utilizes [skip ci] to prevent infinite loops.

---

🎯 Finalized Feature Design (The "Golden" Spec)

1. Context & Goal

Transform documentation into a self-documenting, living portal that stays in sync with the codebase via automated CI/CD pipelines.

2. Documentation Structure

• /docs/architecture/: High-level design, including Mermaid.js diagrams.
• /docs/how-to/: Practical guides for users and agents.
• README.md: The entry point, featuring an embedded, live-updated architecture diagram.
• MCP_REFERENCE.md: An auto-generated, schema-accurate guide for all available MCP tools.

3. CI/CD Workflows

• python-ci (on Pull Request):
  • Linting (Ruff), Type-checking (Mypy), Testing (Pytest).
  • Docstring Coverage Guardrail: interrogate must report $\ge 90%$ coverage for src/cgis.
• autodoc (on merge to main):
  1. Ingestion: Run cgis ingest <path>.
  2. Diagram Gen: Run cgis trace <FQN> --format mermaid.
  3. README Injection: Run scripts/inject_readme_graph.py.
  4. MCP Ref Gen: Run scripts/generate_mcp_ref.py (using asyncio to introspect mcp.list_tools()).
  5. Commit: Push updated README.md and MCP_REFERENCE.md with [skip ci].

4. Automation Scripts

• scripts/inject_readme_graph.py: Uses HTML comment delimiters (<!-- START_CGIS_GRAPH -->) to ensure idempotent updates to README.md.
• scripts/generate_mcp_ref.py: Introspects FastMCP tools and maps their Pydantic schemas to Markdown tables.

5. Verification (TDD)

• Test 1: Docstring coverage guardrail fails CI if coverage < 90%.
• Test 2: inject_readme_graph.py is idempotent.
• Test 3: generate_mcp_ref.py successfully extracts all registered tools and their schemas.