Skip to content

.toml is in no extension set, so config/data TOML is never collected — not even as a document #2434

Description

@wrightbuilt

Summary

.toml appears in none of CODE_EXTENSIONS, DOC_EXTENSIONS, PAPER_EXTENSIONS or IMAGE_EXTENSIONS, so it is absent from _WATCHED_EXTENSIONS and is never collected at all. Unlike .yaml — which at least reaches the semantic/LLM path as a document (#2043, #2214) — TOML has no route into the graph at any cost tier.

Verified on graphifyy==0.9.32:

from graphify.detect import CODE_EXTENSIONS, DOC_EXTENSIONS, PAPER_EXTENSIONS, IMAGE_EXTENSIONS
watched = CODE_EXTENSIONS | DOC_EXTENSIONS | PAPER_EXTENSIONS | IMAGE_EXTENSIONS
".toml" in watched   # False
".yaml" in watched   # True   (DOC_EXTENSIONS, detect.py:32)
".json" in watched   # True   (CODE_EXTENSIONS, and extract_json exists)

.ini and .cfg are in the same position.

Why this is more than a missing extractor

There is a narrow escape hatch — manifest_ingest.is_package_manifest_path() — but it matches five filenames:

PACKAGE_MANIFEST_NAMES = {'apm.yaml', 'apm.yml', 'go.mod', 'pom.xml', 'pyproject.toml'}

So pyproject.toml yields a single package node and every other TOML file yields nothing. Two separable consequences:

1. Cargo.toml is not recognized. Rust's package manifest is missing from a set that already includes Python's, Go's and Maven's. A Rust repo gets no package node where an otherwise-identical Python repo does. That looks unintended, and is a one-line fix independent of the rest of this issue.

2. Config/data TOML is structurally invisible. In our case the affected files are the repository's actual data contracts:

file role nodes
registry.toml repo → role/owner/dependency edges 0
roster.toml team identity 0
tiers.toml per-tier tool enablement 0

Those three files are most of the reason the repository exists. Its graph contains the TypeScript that reads them (827 nodes) and the Markdown that describes them (646 nodes), but nothing they encode. osv-scanner.toml, rust-toolchain.toml, netlify.toml and .cargo/config.toml are in the same class.

Request

Smallest first; any of these helps:

  1. Add Cargo.toml to PACKAGE_MANIFEST_NAMES.
  2. Put .toml (and .ini/.cfg) in DOC_EXTENSIONS so they become reachable, matching .yaml's current status.
  3. A deterministic structural extractor keyed on tables/keys, the way extract_json handles JSON — the generic form of what deterministic ingestion of structured data files #2214 requests.

(2) alone would be a real improvement over the status quo even though it routes to the LLM, because today there is no configuration — API key, flag, or ignore file — that makes these files visible.

Relationship to existing issues

Not a duplicate of #2214, which concerns how structured files are extracted once collected (LLM vs deterministic). This is one step earlier: TOML is never collected. Related to #2043, which addresses the same reachability question for YAML.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions