Language Analysis

Language Analysis & Adding a Language

How PolyGraph turns source files into a GraphModel — and how to teach it a new language. This is the provider boundary seam from the Architecture Overview.

The kernel

lib/kernel/ is decoupled from the UI behind one function:

analyzeProject(files: SourceFileMap, options?) → AnalyzeResult   // lib/kernel/index.ts

What it does:

1. Bucket files by extension and route each bucket to the provider that claims it.
2. Run providers concurrently (Promise.all). The TS/JS provider is synchronous; tree-sitter providers are async (the native core returns a Promise).
3. Merge the providers' fragments: de-duplicate nodes/edges by id, and drop edges whose source or target isn't in the final node set.
4. Isolate failures. A single provider throwing (e.g. a malformed pack query) is caught and surfaced as an error — it never sinks the whole analysis.

The result also carries an unresolved list (references that couldn't be resolved) which flows through to the Problems panel.

Two kinds of provider

The registry (lib/kernel/registry.ts) lists every provider. There are two species:

1. Precise: the ts-morph provider (lib/analyzer/)

For TypeScript/JavaScript (incl. .mts/.cts/.mjs/.cjs, .jsx/.tsx, and .vue/.svelte shells). It uses the TypeScript type checker — so its edges are exact:

File	Emits
imports.ts	import/export edges
calls.ts	type-resolved call edges (the right handle, not just any handle)
components.ts	JSX <Foo/> → renders edges to the component definition
inheritance.ts	extends / implements
composition.ts	has (typed fields, resolved through Array<T> and generics)
externals.ts	external package nodes, enriched from package.json
nodes.ts	symbol/member extraction; roles (React/Vue/Svelte/Angular/ECS)

For very large projects the provider batches the ts-morph Project (a file-count threshold around 8000, batches of ~3000) and forgets nodes between blocks to bound memory — normal-sized repos are unaffected.

2. Declarative: tree-sitter packs (language-packs/ + analyzer-core/)

For the other 25 languages. A pack is purely declarative — no code:

language-packs/<id>/
  pack.yaml    # id, extensions, grammar name, import style
  tags.scm     # a tree-sitter query

tags.scm uses the standard capture convention:

@definition.<kind>  @name                  ; a symbol node
@reference.<rel>    @name                  ; an edge (resolved by name)
@import             @module  [@import.name]; an import edge

The native Rust core runs them.

The native core (analyzer-core/)

A napi-rs addon exposing one function:

async fn analyze(grammar, query, importStyle, filesJson) -> json

• Built with cd analyzer-core && bunx @napi-rs/cli build --release; the prebuilt analyzer-core.node is committed so the app runs without a Rust toolchain.
• Loaded via process.dlopen by absolute path (lib/kernel/treesitter/core.ts) — not require, which webpack would try to bundle.
• Runtime: tree-sitter 0.25 (grammar ABI 13–15). A grammar must depend on the version-agnostic tree-sitter-language crate; one with a direct tree-sitter dep conflicts with the pinned runtime. The core validates each grammar's ABI and rejects stale ones rather than crashing the C layer.
• Hardened: queries are shared via Arc<Query>, and parsing runs under catch_unwind so a single grammar panic can't abort the sidecar.
• WebAssembly text (.wat) has no published grammar crate, so its generated parser.c is vendored in analyzer-core/vendor/wat/ and compiled by build.rs via cc.

How references resolve

• Python uses dotted/relative module resolution against the scanned file set.
• Every other pack language uses by-name resolution against a global unique-definer index (separator-agnostic for . / :: / /), so same-scope references without an explicit import still link. These edges carry inferred confidence (vs. the TS provider's exact).

What becomes a node

In the Rust core, members (method/field/property/…) and containers always become nodes; free functions/variables/constants emit only at the top level (locals fold away). Modules/namespaces are non-absorbing, so items inside a mod stay separate nodes. References attribute to the innermost enclosing emitted symbol.

Adding a language

Because the pack is declarative, adding a language is mostly a new folder:

1. Create the pack — language-packs/<id>/pack.yaml (id, extensions, grammar, import style) and tags.scm (the capture query).
2. Register it — add the id to TREE_SITTER_PACKS.
3. Teach the file filter — add the extensions to lib/file-filters.ts so the scanner picks the files up.
4. Wire the grammar in the core — add "<id>" => Some(tree_sitter_<lang>::LANGUAGE.into()) to language_for in analyzer-core/src/lib.rs, and the crate to Cargo.toml.
5. Rebuild the addon — cd analyzer-core && bunx @napi-rs/cli build --release, and commit the new analyzer-core.node.

A pack can later graduate to a precise, code-backed provider (like ts-morph) without changing any consumer — that's the payoff of the provider boundary. See Development & Building for the build/test loop.