AST Parsing

AST Parsing

codebase-vis uses tree-sitter — incremental, error-tolerant parsers that produce a concrete syntax tree for each language. This is fundamentally different from regex-based dependency extraction: tree-sitter understands grammar, not patterns.

Architecture

The parser system has three layers:

┌─────────────────────────────────────────────────┐
│                 Orchestration                    │
│  parser/index.js                                 │
│  - GRAMMAR_MAP (extension → language)            │
│  - parseFile() - single file                     │
│  - parseFileBatch() - parallel via WorkerPool    │
│  - parserCache (Map<ext, Parser>)                │
└─────────────────┬───────────────────────────────┘
                  │ delegates to
┌─────────────────▼───────────────────────────────┐
│               Language Parsers                   │
│  javascript.js  typescript.js  python.js         │
│  cpp.js         java.js        go.js             │
│  rust.js        html.js        css.js            │
│  Each exports:                                   │
│  - grammar                                       │
│  - extractDependencies(node) → string[]          │
│  - extractEntities(node) → { classes, functions, │
│    methods, docstrings }                         │
└─────────────────┬───────────────────────────────┘
                  │ used by
┌─────────────────▼───────────────────────────────┐
│                  Utilities                       │
│  languages.js  - LANGUAGES, EXT_TO_LANGUAGE,     │
│                  KNOWN_EXTENSIONS, STACK_MARKERS │
│  stack-detector.js - detectTechStack()           │
│  parse-worker.js - fork()-ed child process       │
└─────────────────────────────────────────────────┘

Supported Languages

Language	File Extensions	Parser Grammar
JavaScript	.js, .jsx	tree-sitter-javascript
TypeScript	.ts, .tsx	tree-sitter-typescript (TS + TSX)
Python	.py	tree-sitter-python
C / C++	.cpp, .h, .hpp	tree-sitter-cpp
Java	.java	tree-sitter-java
Go	.go	tree-sitter-go
Rust	.rs	tree-sitter-rust
HTML	.html	tree-sitter-html
CSS	.css	tree-sitter-css

Dependency Queries

Each language module defines tree-sitter S-expression queries that capture real import/require/include statements:

Language	Query Targets	Examples Captured
JavaScript	import_statement, call_expression (require + import())	import { x } from 'y', const x = require('y')
TypeScript	Same as JS (both .ts and .tsx grammars)	ES imports, require(), dynamic imports
Python	import_statement, import_from_statement	import os, from .foo import bar
C / C++	preproc_include (system + string literals)	#include <vector>, #include "my.h"
Java	import_declaration	import java.util.List
Go	import_spec	import "fmt"
Rust	use_declaration, extern_crate_declaration	use std::collections::HashMap
HTML	element with src/href attributes	<link href="...">, <script src="...">
CSS	import_statement, url() call expressions	@import "file.css", background: url("img.png")

Entity Extraction

Each file is scanned for entities — classes, functions, methods, and docstrings — using dedicated tree-sitter queries per language.

Example: Python Method Query with Decorators

(query
  (class_definition body: (block (function_definition name: (identifier) @method_name)))
  (class_definition body: (block (decorated_definition definition: (function_definition name: (identifier) @method_name))))
)

This captures both regular methods and decorated methods (@property, @staticmethod, etc.). Methods are excluded from top-level function lists via AST position key deduplication to avoid double-counting.

Entity Types

Entity	Description	Example
classes	Named class/struct/interface declarations	class Foo, struct Bar
functions	Top-level function declarations	function foo()
methods	Functions defined inside classes	class { method() }
docstrings	Documentation comments (/** */, """ )	JSDoc, Python docstrings

Parser Caching

Parsers are instantiated once per file extension and cached in a Map to avoid re-initializing tree-sitter grammars for every file:

const parserCache = new Map()

function getParser(ext) {
  if (parserCache.has(ext)) return parserCache.get(ext)
  const parser = new Parser()
  parser.setLanguage(GRAMMAR_MAP[ext]().grammar)
  parserCache.set(ext, parser)
  return parser
}

Known Inconsistency

The Rust parser (src/parser/rust.js) returns entities as a flat string[] array, while all other language parsers return { classes, functions, methods, docstrings }. The graph builder handles both formats for backward compatibility.