Skip to content

v1.108.236 - a TOML table no longer claims the next table's header line

Choose a tag to compare

@jgravelle jgravelle released this 04 Aug 14:28
· 400 commits to main since this release

end_line for a TOML table ran one row past the table's own text, onto the line holding the next table's header. Against fastapi/fastapi at a64dfbbd, [build-system] (lines 1-3, blank at 4) reported end_line=5 — the line holding [project], a different symbol entirely. [tool.mypy] reported 206, the line holding [[tool.mypy.overrides]].

Cause, which is not TOML-specific

A tree-sitter node whose end_point sits at column 0 stops at a line boundary and holds no text on that row, so the end_point[0] + 1 idiom overshoots by one. It surfaces here because tree-sitter-toml runs a table node to the start of the following table, putting the boundary exactly on another symbol's header.

byte_length was correct throughout. The two fields disagreed and only the line number was wrong, which is why nothing reading the body ever noticed. A test now pins their agreement, so drift in either fails rather than being absorbed by the other.

Scope was measured, not assumed

73 sites in parser/extractor.py share the idiom and none carried a guard. Sampled across two corpora, Python, Go, TypeScript, JavaScript, JSON and CSS showed zero disagreement, because their symbol nodes end at the last token. Only the three TOML sites changed; sweeping all 73 would move line numbers for languages with no demonstrated defect.

Known issue, deliberately not addressed here

A separate YAML discrepancy affects a different field: there end_line is right and byte_length reaches past the symbol. health-radar-comment.yml::name holds 27 bytes of text recorded as 28; a sampled step key recorded 74 bytes covering two sibling keys. Different subsystem and a worse blast radius (an over-long extent puts content in a body that is not the symbol's), so it gets its own investigation rather than a footnote in a line-number fix.

Upgrade note

No INDEX_VERSION bump, so existing indexes keep the old value until the file is re-parsed. Re-index a file, or let the watcher do it, to pick up the correction.

Reachable from bounded-mode source_end_line clamping, get_file_outline spans, and diff-to-symbol mapping. Verified at the caller's entry point rather than only at the parser: 119 TOML symbols walked through get_symbol_source on a freshly built index, zero line-span disagreements, against 11-of-40 before. Tests proven non-vacuous against the old expression.

Verification status: full suite 6834 passed / 7 skipped / 1 failed. The single failure is test_v1_108_207::test_warms_sentence_transformers, which fails identically on the unmodified previous release commit: it is environment-coupled and goes red on any machine without the optional sentence_transformers dependency. It is not related to this change and is not a regression.