Entl

Entl turns specific kinds of external data into typed Rust facts that can be inspected, shaped, and moved without first forcing them through a database or a stringly typed table interface.

The first domain is a source codebase's present tree. entl-codebase walks a local checkout once and returns reusable facts about:

• files, language-detection evidence, and lexical comment syntax;
• Cargo and package.json packages;
• Cargo and JavaScript workspace membership;
• package managers inferred from declarations, workspace membership, and lockfiles;
• project boundaries, ecosystem roles, and language project signals;
• language-linked test-layout, inline-test, and required-config conventions;
• language-linked tool profiles and typed task classifications;
• direct dependency names, kinds, declared sources, and version requirements;
• exact package versions, registry sources, and checksums observed in Cargo lockfiles;
• package scripts;
• distributable artifact instances for binaries, site bundles, napi-rs addons, and Tauri apps;
• the nearest package owners of every file; and
• recoverable inventory diagnostics.

Explicit compiler observation is separate from passive inventory. Consumers that need build-context facts can call observe_rust_compiler; it runs the active rustc from the repository directory and returns its exact release, commit, host, sysroot, installed standard-library source location, cfg values, and target features. A normal inspect never runs a compiler.

The crate enforces no codebase policy. A linter can consume files and lazy text, an auditor can consume package/workspace structure, and a later tree-sitter adapter can turn identified source files into symbols and syntax facts.

entl-github derives provider-specific facts from that reusable codebase inventory. It recognizes GitHub Actions workflow files and triggers, expands package scripts, and uses entl-codebase tool profiles to produce typed test, lint, format, typecheck, and build invocations. It does not decide which tasks policy requires. Build invocations retain typed artifact outputs and exact package scopes, including Cargo workspace members.

There are two entry points. walk returns only the file layer—paths, sizes, language evidence, diagnostics, and lazy content reads. inspect builds the package and workspace facts on top of the same walk. Linked consumers can add deterministic enrichment handlers through discovery_registry.

GitHub inspection is an explicit second step:

let codebase = entl_codebase::inspect(".", &entl_codebase::InventoryOptions::default())?;
let github = entl_github::inspect(&codebase);

for workflow in &github.workflows {
    println!("{}", workflow.path.display());
}
# Ok::<(), entl_codebase::Error>(())

use entl_codebase::{InventoryOptions, inspect};

let codebase = inspect(".", &InventoryOptions::default())?;

for package in &codebase.packages {
    println!("{} at {}", package.id, package.root.display());
    for file in codebase.files_for_package(&package.id) {
        if let Some(language) = &file.language {
            println!("  {} ({})", file.path.display(), language.language);
        }
    }
}
# Ok::<(), entl_codebase::Error>(())

A source scanner can avoid manifest parsing:

use entl_codebase::{InventoryOptions, walk};

let tree = walk(".", &InventoryOptions::default())?;
for file in tree.files_with_language("rust") {
    let source = tree.read_text(&file.path)?;
    println!("{}: {} bytes", file.path.display(), source.len());
}
# Ok::<(), entl_codebase::Error>(())

Runtime Tree-sitter parsers

entl-tree-sitter loads versioned Wasm parser packs at runtime. Grammar implementations are data artifacts rather than Rust dependencies. Verified Rust, JavaScript, TypeScript, and TSX packs are included under parser-packs. Each manifest declares which Entl language and file extensions it handles, so multiple grammar variants can serve one language without consumer hardcoding. Parser packs also declare language-specific syntax-unit node kinds used to group token comparisons at function, method, implementation, and class boundaries.

The grammar.wasm artifacts are vendored third-party builds, not Entl's own code. Each pack's parser.toml pins the upstream repository, revision, version, license, and the artifact's sha256. Upstream licenses are reproduced in THIRD-PARTY-NOTICES.md.

A pack may also ship Tree-sitter queries as queries/*.scm, named by file stem, which is the convention the wider ecosystem uses and what makes an upstream grammar's own queries usable as vendored. A pack carries the query text; compiling needs a loaded grammar, so ParserRuntime::load compiles them and a query that does not compile fails the load. That is deliberate: a broken query matches nothing, and a consumer that matches nothing reports nothing, which is indistinguishable from source that is genuinely clean.

LoadedParser::matches runs a query over a parsed file and returns each match with its captures named. Naming a query the pack does not ship is an error rather than an empty result, for the same reason. Queries have no negation, so a pattern says "nothing here" by marking a capture optional and letting it be absent from the match — QueryMatch::has is how a consumer reads that:

((match_arm pattern: (match_pattern
   (tuple_struct_pattern type: (identifier) @variant (identifier)? @bind))) @arm
 (#eq? @variant "Err"))

Err(_) matches without @bind; Err(e) matches with it. ParseProvenance carries a queries_sha256 alongside the grammar digest, because a fact derived through a query depends on that query's text as much as on the grammar, and two runs whose queries differ would otherwise be indistinguishable.

Workspace

crates/entl-codebase   typed codebase inventory and profiles
crates/entl-github     typed GitHub workflow and automation facts
crates/entl-semantics  span-anchored semantic observations, language neutral
crates/entl-rust-mir   observes resolved Rust semantics by running as the compiler
crates/entl-tree-sitter runtime-loaded Wasm parser packs
parser-packs           pinned runtime grammar artifacts (third-party, vendored)
tools/verbosity regenerates the language verbosity table from a corpus
docs/design.md         boundaries and planned adapters

This is intentionally not yet a CLI, query engine, database, binding suite, or generic ETL framework. See the design.

Semantic observations

Syntax cannot say where a call goes. use std::fs; fs::read(path) and std::fs::read(path) are one call written two ways, and only name resolution knows it. entl-semantics defines what a compiler or language server can be asked about a place in the source: what a name refers to, what type an expression has, where a call goes, what a type implements. Every observation is optional, and Coverage records which questions a provider attempted, so a consumer can tell "nothing found" from "not looked at".

The schema deliberately holds no intermediate representation. Compilers disagree at that level — some expose a control flow graph, some a typed syntax tree, some neither — and unifying those yields something less useful than any of them. Unifying the answers they can all give does not.

entl-rust-mir is the first provider. It replaces rustc for one compilation and reads the resolved mid-level representation:

cd crates/entl-rust-mir && cargo build
ENTL_RUST_MIR_OUTPUT=/tmp/observations \
  target/debug/entl-rust-mir --crate-type lib --crate-name mycrate src/lib.rs

It lives outside the workspace on a pinned nightly with the compiler's private crates. That isolation is the point: compiler-backed observations sharpen results where a toolchain is available, and Tree-sitter remains the floor everywhere else. A language whose compiler is not integrated still parses.

Language verbosity

Language profiles carry a measured fact about how much source text a language needs: LanguageProfile::verbosity for one language against the baseline, and verbosity_ratio for a pair as it was actually measured.

use entl_codebase::{language_profile, verbosity_ratio};

let java = language_profile("java").unwrap().verbosity().unwrap();
let python = language_profile("python").unwrap().verbosity().unwrap();
assert!(java.bytes > python.bytes);

// Measured on the exercises both implement, not derived from the two indexes.
let measured = verbosity_ratio("java", "python").unwrap();
assert!(measured.tasks > 100);

The numbers come from comparing Entl's languages on a corpus of the same task solved in each of them, on the units each pair has in common. Because no two pairs share a unit set, the ratios are not transitive, and the single index per language is a fit rather than a fact — each profile reports how far off that fit gets.

The shipped table is measured on Exercism, where each exercise has one reference solution per track written against a shared specification and test suite. Two other corpora are published alongside it as cross-checks. Rosetta Code is ten times larger and entirely uncontrolled. mal is one mid-sized program — a Lisp interpreter of one to five thousand lines — implemented in all sixteen languages against one test suite.

All three rank the languages alike (Spearman 0.79 to 0.87) and disagree on magnitude. The disagreement has a direction: the languages spread 2.5x apart on Exercism's small exercises and 4.9x apart on mal's mid-sized program, because a small exercise measures mostly the absence of ceremony while a real program has structure every language must pay for. Treat the shipped index as a property of a corpus, not of a language, and read docs/verbosity-mal.md before carrying it to anything program-sized.

Verbosity here is also not a porting factor. It measures what a language needs when someone writes a solution from the specification. Translating an existing program is a different quantity and can run the other way: Bun's Zig-to-Rust rewrite expanded by about 1.37x in tokens, while independently written Rust and Zig solutions put Rust below Zig.

Regenerate with tools/verbosity; no corpus is redistributed here, only the measurements. notes/verbosity.md is the analysis of record: what the corpora disagree about, why the index is a ranking device rather than a constant, and which candidate corpora were rejected and on what evidence.

Development

cargo fmt --all --check
cargo test --workspace
cargo clippy --workspace --all-targets -- -D warnings

License

Entl's own source is MIT. The vendored Tree-sitter grammars under parser-packs/ are third-party works under their own licenses; see THIRD-PARTY-NOTICES.md.