Deliberately-skipped data JSON (#1224) is classified as failed extraction, so it is re-queued on every incremental run

[GridWork-dev]

Summary

extract.py's empty-result classifier cannot distinguish "the extractor deliberately declined this file" from "extraction failed". Data-shaped JSON is intentionally not AST-walked (#1224 — it is left to the LLM semantic pass), but because a .json extractor is registered, every such file trips the failed_sources path, never gets frozen as processed in the incremental manifest, and is therefore re-queued on every subsequent run, forever.

Correctness is unaffected. The cost is unbounded repeated work on incremental runs.

The mechanism

extract.py classifies a file as failed when it produced no nodes and an extractor exists for it:

if (not _res.get("nodes")) and _get_extractor(_p) is not None:
    if _key not in _failed_seen:
        _failed_sources.append(_key)
        _failed_seen.add(_key)

failed_sources is then returned with the explicit purpose (per the comment in the same file) of keeping those paths out of the processed set:

# Surfaces failed/empty AST sources to the CLI so the incremental
# manifest does not freeze them as processed (#2543).
"failed_sources": _failed_sources,

For data JSON both conditions are true by design: _get_extractor() returns the JSON extractor because the extension is registered, and the extractor correctly returns no nodes because the file is data rather than a config/manifest. Zero nodes here is the intended outcome, not a failure — but it is indistinguishable from one at this call site.

Reproduction

Any repository containing data-shaped JSON alongside config JSON. On a 3,329-file repo indexed with 0.9.47:

manifest entries:                                3329
.json with BLANK ast_hash (re-queued every run):  110
.json with populated ast_hash:                    178
blank ast_hash, all extensions:                   110
blank by extension: [('.json', 110)]

Every entry with a blank ast_hash is a .json file; no other extension is affected, which matches the classifier path above rather than any general extraction problem. Examples are Fumadocs-style meta.json sidecars:

apps/site/content/docs/agentic-dev/meta.json
apps/site/content/docs/ai-production/meta.json
apps/site/content/docs/base/meta.json

These are stable files that have not changed in many runs, yet are re-processed on each incremental pass. With a 15-minute indexing cadence this repeats indefinitely.

Suggested direction

The extractor needs a way to report "declined by design" separately from "produced nothing", so the classifier can exempt the former. Concretely, either:

1. have the JSON dispatch return an explicit marker (e.g. {"nodes": [], "edges": [], "declined": True}) and skip failed_sources when it is set; or
2. have _get_extractor() return None for JSON files the structural extractor has already determined it will not walk, so the existing is not None guard does the right thing.

(1) is probably less invasive since _get_extractor() is dispatch-by-extension and does not currently inspect content.

I am happy to open a PR if a preferred shape is indicated.

Environment

• graphifyy 0.9.47 (PyPI), Python 3.14, macOS 15 arm64
• Incremental runs via graphify extract <src> --backend openrouter --out <dir>

[safishamsi]

Fixed in v0.9.48: https://github.com/Graphify-Labs/graphify/releases/tag/v0.9.48 (a deliberately-declined data JSON is no longer counted as a failed extraction, so it stops being re-processed every run). Closing as resolved.