Skip to content

gotreesitter v0.58.0

Latest

Choose a tag to compare

@github-actions github-actions released this 20 Nov 18:33
· 2 commits to main since this release
4ab5d4e

Highlights

  • 34 grammars now parse by default through ccore, the C-faithful parser
    core. Each passed the graduation gate in its pull request: trees and subtree
    reuse equal pinned C, and time and memory at or below the existing engine.
  • Other grammars keep the existing engine, and calling code does not change.
    parser.SetAdmissionCandidateRoute(false) keeps one parser off the core.
  • Add LangEntry.ParseContext and LangEntry.ParseStrict for cancellation and
    early-stop errors, and Tree.RetryStopReason().

Upgrade notes

  • Trees for the 34 certified grammars can change where v0.57.0 differed from
    the C runtime, for example on text with syntax errors; they now match C. This
    affects callers that compare trees or node counts across versions. To keep
    the v0.57.0 engine for one parser, call
    parser.SetAdmissionCandidateRoute(false); it takes effect on the next
    parse, and a tree the core built is then reparsed fresh.
    ClearAdmissionCandidateRoute() returns the parser to the default. The
    process-wide SetAdmissionCandidateRouteDefault and
    GTS_ADMISSION_CANDIDATE do not move certified grammars off the core.

Added

  • Add LangEntry.ParseContext(ctx, src) for cooperative context
    cancellation and deadlines. Existing entry.Parse(src) calls remain supported
    (#1590).
  • Add LangEntry.ParseStrict(ctx, src) to report early stops while preserving
    registered token-source selection. Non-nil trees remain caller-owned even
    when an error is returned. Both methods use the registered token source when
    there is one. Cancellation sets the parser's existing cancellation flag, so it
    cannot interrupt a blocking loader, scanner or token source, and a parse that
    finished before a late cancellation keeps its result
    (#1590).
  • Add internal/ccore, a C-faithful parser core: parser, stack, subtrees,
    lexer, error recovery, external scanners, edits and reuse, using only the
    standard library. It has no public API of its own; certified grammars reach
    it through the ordinary Parser (see Changed). On the test corpus it equals
    pinned C 0.27.1 on 1,197 of 1,198 decision traces, 1,283 of 1,283
    visible-tree digests, 1,233 of 1,233 record sets and 509 of 509 incremental
    session steps. On real files, 178 of 202 grammars are fully equal
    (#1592,
    #1595).
  • Add root adapters that build ccore tables from a *Language, the packages
    internal/ccore/bind and internal/ccore/bind/bindtest, table validation,
    and a required CI job, ccore-equality-cgo. On every pull request that
    changes code, the job compares the new core with pinned C for json, python, c
    and typescript
    (#1595).
  • Add Tree.RetryStopReason() and the ParseRuntime.RetryStopReason field. They
    report ParseStopTimeout or ParseStopCancelled when a timeout or the
    cancellation flag kept the parser from finishing its error-recovery retry and
    fallback work, and ParseStopNone otherwise. ParseStopReason() and the
    returned tree are unchanged: the stop reason stays the one of the pass that
    produced the tree, usually ParseStopAccepted
    (#1600).

Changed

  • Serve 34 grammars through ccore by default. A grammar is served there
    only when its exact blob is in testdata/ccore_certificates.json, and only
    for the parse modes its certificate covers. A parse still uses the existing
    engine when it sets included ranges, a custom token source, parse work
    limits (SetParseWorkLimits) or parse-observability hooks. The grammars, by
    pull request:
  • Work counters for the certified grammars change: a fresh parse lexes the
    tokens C lexes, and an edit that starts from a malformed tree reparses fresh
    instead of reusing subtrees. Each grammar's pull request lists its
    counter-ledger rows and the reason for each.
  • Run the editor-latency CI gate as one job per language, which cuts a pull
    request's CI from about two hours to about 26 minutes, and skip it for pull
    requests that change only internal/ccore, its harness tests or
    documentation (#1594).
  • Make the cgo harness repair file ownership after its Docker runs, so files
    that a root container writes into a mounted folder no longer block later
    non-root builds (#1589).

Fixed

  • Compact projection: keep raw cumulative dynamic precedence through hidden
    projection, unary reductions and retained aliases, so checked additions match
    native int32 boundary behavior (#1588).
  • Grammar generation: fall back to LALR when precise item-set construction
    exhausts its state budget. The old path kept the incomplete automaton, which
    gave an ERROR root for valid input in the regression fixture. If the
    fallback also exhausts its budget, generation returns an error instead of
    tables (#1585).
  • ts2go: stop reading the index in a designated C supertype initializer as an
    entry, which shifted the subtype table. Unsupported entries now return an
    error instead of partial metadata. This does not update the F# grammar
    (#1584).
  • Work-count diagnostics: make attempt counters and phase breakdowns
    exclusive, avoiding false Overflow reports caused by counting nested work
    twice. This applies only to gts_workcount builds
    (#1591).
  • Editor-latency gate: fix the preparation of the baseline module when shared
    harness tests import packages that the baseline lacks. Packages that the
    timing driver needs must still build
    (#1595).
  • Norg: the scanner ends verbatim-tag and paragraph-delimiter tokens where
    the pinned C scanner does and counts every detached marker it reads, so a
    5,382-byte document that used to stop with an error at byte 381 now parses
    to the end and matches C's tree
    (#1674).
  • Generated lexers: keep named string keywords as their own tokens
    (#1642).
  • TypeScript: preserve member names and generic fields (#1604).
  • JavaScript: keep the declarations that follow a stray quote (#1701).
  • Incremental parsing with a custom token source: retry a failed reparse from
    fresh input instead of returning the failed tree (#1702).
  • Parser: give each GLR stack its own lex mode for the shared lookahead, and
    revive a stack whose lex mode read a wider lookahead (#1629, #1632); stop
    no-action re-lex steps from undoing each other (#1630).
  • Result trees: report error extras left beside the result root (#1635).
  • Make: an escaped comma stays inside a word (#1611).
  • RootNodeWithOffset and injected trees add points the way C's
    point_add does (#1614).
  • Legacy engine: bound forest searches by child end byte (#1703), and synchronize
    parser switches and arena profiling so the race detector stays quiet (#1700).
  • ts2go: read character-range arrays strictly and fail closed (#1605).
  • Bound the pooled scratch that a tiny parse can receive (#1613).
  • External scanners: fourteen ports now do what their C scanners do on unusual
    input: a no-break space or a line separator where whitespace is expected, a
    NUL byte, or an invalid UTF-8 byte. Before, the
    ports of tablegen, nushell, godot_resource, julia, ron, lua, fennel, tcl,
    markdown_inline, gitcommit, powershell, luau, odin and fish could return
    another token than the C runtime on such input, for example a PowerShell
    statement terminator at U+00A0, or look at more or fewer characters
    (#1658).
  • External scanners: nine more ports now do what their C scanners do on
    unusual input: an embedded NUL byte, a non-ASCII space, letter or digit where
    C accepts ASCII only, or an invalid UTF-8 byte. Before, the ports of erlang,
    dockerfile, dart, kotlin, nim, r, c_sharp, markdown and ruby could return
    another token than the C runtime on such input, for example a Kotlin
    automatic semicolon, miss a Dockerfile heredoc marker that follows
    whitespace, or look at more or fewer characters
    (#1660).

Evidence

  • Validation and measurement scope are recorded in the linked pull requests,
    including #1592 and
    #1595 for the ccore
    comparison with pinned C.
  • A grammar joins the certificate list only after the one-grammar graduation
    gate (#1663, #1666; docs/v1-design.md). Each graduation pull request under
    Changed reports its results: correctness and reuse against pinned C, timing
    at 32 KiB, 137 KiB and 1 MiB, memory, race, and its counter-ledger changes.
    The CI job ccore-equality-cgo covers json, python, c and typescript; the
    other grammars rely on their own gate.
  • Some graduation pull requests also timed a fresh parse and release of one
    file of about 2 KiB against pinned C in paired Go–C–C–Go cycles over 20
    seeds (the method is in the linked runs): JSON 0.43×, Python 0.44× and Diff 0.30× of C's time
    measured at f3e39124 and 2b4f6885
    (runs),
    and Kotlin 0.38× on its graduation branch
    (runs).
    These four are not a measurement of the release head or of the other
    grammars.
  • TypeScript editing on the locked 19,950-byte W5 file, paired 20 seeds against
    the base before #1675: one-byte, 100-byte and typing edits went from 149×,
    188× and 255× of pinned C's time to 0.83×, 0.89× and 0.85× (about −99.5%)
    (W5 job).