Skip to content

Feature: PageRank Repo Map — Automatic Codebase Context Selection via Symbol Graph (inspired by Aider) #535

Open
Open
Feature: PageRank Repo Map — Automatic Codebase Context Selection via Symbol Graph (inspired by Aider)#535
Labels
type/featureNew feature or request

Description

@teknium1
teknium1
opened on Mar 6, 2026

Overview

Aider's most powerful system is its PageRank-based repo map — an automatic context selection mechanism that builds a directed graph of symbol definitions and references across an entire codebase, then uses personalized PageRank to rank files by relevance to the current conversation, and renders the top-ranked definitions as scope-aware elided code views that fit within a token budget.

When Hermes Agent works on code, it relies on the user or agent to manually read_file and search_files to find relevant context. There's no automatic awareness of codebase structure — the agent has to guess which files matter. Aider solves this: before every LLM call, it computes which symbols and files are most relevant to the files being edited and includes a compact structural overview in the prompt. This is fundamentally different from both regex search (#489's semantic search via embeddings) and explicit @ references (#502) — it's automatic, graph-based, zero-effort context that adapts to the conversation.

Source: Deep analysis of aider/repomap.py (781 lines), tree-sitter query files, and integration with the Coder classes.

Research Findings

How Aider's Repo Map Works (Step by Step)

The algorithm has 6 phases:

Phase 1: Symbol Extraction (tree-sitter)

For each file in the repo, Aider:

  1. Determines the language from file extension
  2. Loads a tree-sitter parser and .scm query file (stored in aider/queries/)
  3. Parses the file into an AST
  4. Runs tag queries to extract captures: @name.definition.X → definitions, @name.reference.X → references
  5. Yields Tag(rel_fname, fname, line, name, kind) namedtuples

Fallback: If tree-sitter queries only produce definitions (e.g., C++ queries lack reference captures), Aider uses Pygments lexer tokenization to extract all Token.Name tokens as references.

Tags are cached in SQLite (via diskcache.Cache), keyed by filename with mtime-based invalidation — so re-parsing only happens for changed files.

Phase 2: Graph Building

Builds a networkx.MultiDiGraph where nodes are files and edges go from referencing file → defining file, weighted by reference importance:

# Edge weight calculation (simplified)
mul = 1.0
if ident in mentioned_idents:     mul *= 10   # user mentioned this symbol
if is_meaningful_ident(ident):    mul *= 10   # snake_case/camelCase, >=8 chars
if ident.startswith("_"):        mul *= 0.1  # private symbol discount
if defined_in_more_than_5_files: mul *= 0.1  # too common = less relevant

# CRITICAL: 50x boost for references FROM files currently being edited
if referencer in chat_fnames:     mul *= 50

# Dampen high-frequency references
num_refs = math.sqrt(num_refs)

G.add_edge(referencer, definer, weight=mul * num_refs, ident=ident)

The 50x multiplier for chat-file references is the key insight: it ensures that symbols used by the files you're editing get ranked highest.

Phase 3: Personalized PageRank

personalize = {fname: 100 / len(fnames) for fname in chat_fnames}
# Also boost files mentioned in user's message, files matching mentioned identifiers

ranked = nx.pagerank(G, weight="weight", personalization=personalization,
                     dangling=personalization)

This captures transitive importance — a utility file used by many important files ranks high even if not directly mentioned.

Phase 4: Distribute File Rank to Definitions

Each file's PageRank score is distributed to (file, symbol) pairs proportionally to edge weights:

for src in G.nodes:
    src_rank = ranked[src]
    total_weight = sum(data["weight"] for _, _, data in G.out_edges(src, data=True))
    for _, dst, data in G.out_edges(src, data=True):
        ranked_definitions[(dst, data["ident"])] += src_rank * data["weight"] / total_weight

Files already in the chat context are excluded (they're already fully visible).

Phase 5: Binary Search for Token Budget

Aider does binary search over the ranked definition list to find the maximum number of entries that fit the token budget (default: max(1024, min(max_input_tokens/8, 4096)) tokens):

while lower_bound <= upper_bound:
    tree = self.to_tree(ranked_tags[:middle], chat_rel_fnames)
    num_tokens = self.token_count(tree)
    if abs(num_tokens - max_map_tokens) / max_map_tokens < 0.15:  # within 15%
        break
    # binary search adjustment...

When NO files are in the chat, the budget increases to 8x (more map since there's no file content).

Phase 6: Scope-Aware Tree Rendering

Uses grep_ast's TreeContext to render each file with intelligent elision:

  • For each ranked symbol, shows its definition line plus parent scope headers (class signature, function signature)
  • Everything else is elided with ⋮ markers
  • Small gaps between shown lines are filled in

Output format:

path/to/file.py:
│class MyClass:
⋮
│    def important_method(self, arg):
⋮

path/to/other.py:
│def utility_function(x, y):
⋮

How It's Sent to the LLM

Injected as a user message with this framing:

Here are summaries of some files present in my git repository.
Do not propose changes to these files, treat them as *read-only*.
If you need to edit any of these files, ask me to *add them to the chat* first.

The map is rebuilt before every LLM call, so it adapts as the conversation evolves.

Why It's Effective

  1. PageRank captures transitive importance — files referenced by many important files rank high even if not directly mentioned
  2. Personalization adapts to conversation — biases toward user's active files
  3. 50x multiplier for chat-file references — symbols used by files being edited get prioritized
  4. Scope-aware rendering — shows just signatures and structure, not full implementations
  5. Adaptive budget — 8x more context when no files selected, shrinks as files are added
  6. Smart heuristics — penalizes private symbols, boosts meaningful identifiers, dampens common symbols

Current State in Hermes Agent

What we have:

The gap: No automatic codebase structure awareness. The agent doesn't know what files exist, how they relate to each other, or which symbols are relevant to the current task — unless it manually searches. For a coding agent, this is a critical missing capability.

Implementation Plan

Classification

This should be a core codebase change — a new tool or enhancement to existing tools. It requires:

  • Tree-sitter AST parsing (native library)
  • Graph computation (PageRank)
  • Token budget management
  • Integration with the prompt assembly pipeline

What We'd Need

  1. Tree-sitter grammars — Python, JS/TS, Rust, Go, C/C++, Java, Ruby, etc. (Aider supports 26+ languages via tree-sitter-language-pack)
  2. Tag query files (.scm) — Language-specific queries for extracting definitions and references (could adapt Aider's, they're Apache 2.0)
  3. Graph library — networkx for PageRank, or a lightweight custom implementation (power iteration is ~50 lines)
  4. Scope-aware renderer — For elided code display (could build on tree-sitter AST, or port grep_ast's TreeContext)
  5. Tag cache — SQLite or diskcache for mtime-based invalidation
  6. Integration point — Either a new repo_map tool, or automatic injection into system prompt when working in a git repo

Phased Rollout

Phase 1: Symbol Extraction + Flat Listing

  • Tree-sitter parsing for top 5 languages (Python, JS/TS, Rust, Go, C/C++)
  • Extract definitions (classes, functions, methods, constants)
  • Cache in SQLite with mtime invalidation
  • New codebase_map tool that returns a flat listing of symbols, ranked by file proximity to specified files
  • No PageRank yet — simple heuristic ranking (same directory → higher rank)

Phase 2: PageRank Ranking + Scope-Aware Display

  • Build reference graph (definitions + references)
  • Implement personalized PageRank (can use networkx or custom)
  • Scope-aware rendering with parent scope headers and elision
  • Token budget fitting via binary search
  • Auto-inject repo map when agent is working in a git repo

Phase 3: Deep Integration

Pros & Cons

Pros

Cons / Risks

  • Tree-sitter dependency — native library that needs compilation for each platform. However, tree-sitter and tree-sitter-language-pack are available as pip packages with pre-built wheels
  • Complexity cost — adds a new subsystem (graph building, PageRank, rendering) with ~600 lines of core logic
  • Token budget tradeoff — repo map tokens compete with conversation context. Default of max_input/8 may be too much or too little depending on the task
  • Large repo performance — PageRank on repos with >50k files may be slow (Aider has RecursionError protection for this case)
  • Not useful for non-code tasks — this is a coding-specific feature, irrelevant when Hermes is used for research, creative work, etc.

Open Questions

  • Should the repo map be a standalone tool (agent decides when to call it) or automatically injected into every prompt when working in a git repo?
  • Should we use networkx (established, 20MB dep) or implement power iteration directly (~50 lines)?
  • What token budget default makes sense? Aider uses max_input/8 — is that right for Hermes which has different context management (probing, compaction)?
  • Should the map be visible to the user (shown in output) or invisible (injected into system prompt)?
  • Can we reuse Aider's .scm query files directly (Apache 2.0 licensed)?

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/featureNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions