Fix unclosed-inline error misclassification + supporting infrastructure (quotes + emphasis)

[rundel]

Apologies in advanced for all of the non-sense that is about to follow - everything started with good intentions and a minor issue around error locations for unclosed emphasis markers:

$ printf -- 'The "_blank" word.\n' | cargo run --bin pampa -- 2>&1

Error: [Q-2-11] Unclosed Double Quote
   ╭─[ <stdin>:1:13 ]
   │
 1 │ The "_blank" word.
   │     ┬┬
   │     ╰─── This is the opening quote mark.
   │      │
   │      ╰── I reached the end of the block before finding a closing '"' for the quote.
───╯

How hard could that possibly be to fix?

It got very ugly very quickly - I'm not sure what follows is a good solution or not but given it took a lot of iteration to even get close to something that mostly worked I think it's worthwhile documenting it all and passing it off. At a minimum it will hopefully provide a partial starting point for a better / cleaner solution.

The commits document a lot (but not all) of the iteration (e.g. adding an additional key to the corpus lookup table) that were eventually scraped. So it may also be useful to have claude summarize those for you individually.

---

Claude's write-up (w/ light editing)

Summary

This branch fixes a class of misleading parse errors where an unclosed
emphasis or quote marker nested inside another inline scope was reported as
if the outer scope were unclosed. The fix evolves through nine commits
that progressively generalize the mechanism, refactor it for
maintainability, and improve the user-facing diagnostic. Along the way it
picks up two unrelated parser fixes that surfaced during the work.

Final shape of the change:

• A scope-stack walker reconstructs the parser's nesting context at error
  time so the diagnostic can target the correct inline element regardless
  of how deeply it was nested.
• The "Unclosed X" diagnostics now anchor their indicator at the closing
  delimiter of the enclosing scope (when one exists) instead of floating at
  the parser-failure position past end of line.
• The message wording adjusts to context: "end of the block" at block
  level, "end of the inline scope" when nested.
• Dispatch is driven by a corpus-annotated unclosedScope field rather
  than any hardcoded Rust mapping or per-(state, sym) corpus authoring.
  Adding a new inline scope kind is a one-line corpus edit plus a walker
  enum variant.

Diff size: 20 files, +2,883 / −466. Most of the insertion volume is the
new outer_scope.rs module (740 LOC, ~half of it tests), the expanded
test_emphasis_in_quote.rs regression suite (61 tests), and the
regenerated _autogen-table.json. The actual implementation logic is ~400
LOC.

Before / after on concrete inputs

Case 1 — Unclosed emphasis between paired double quotes

printf 'The "_blank" word.\n' | cargo run --bin pampa --

Before (on main):

Error: [Q-2-11] Unclosed Double Quote
   ╭─[ <stdin>:1:13 ]
   │
 1 │ The "_blank" word.
   │            ┬┬
   │            ╰─── This is the opening quote mark.
   │             │
   │             ╰── I reached the end of the block before finding a closing '"' for the quote.
───╯

After:

Error: [Q-2-5] Unclosed Underscore Emphasis
   ╭─[ <stdin>:1:12 ]
   │
 1 │ The "_blank" word.
   │      ┬     ┬
   │      ╰──────── This is the opening '_' mark.
   │            │
   │            ╰── Reached the end of the inline scope before finding a closing '_' for the emphasis.
───╯
ℹ If you want a literal underscore, escape it with `\_`.

The diagnostic now points at the unclosed _ rather than sending the user
hunting for a missing " that's already in the input. The indicator is
anchored at the closing " because the parser actually gave up at that
point (the _ couldn't close before the " arrived). A hint suggests the
escape if the user meant the marker literally.

Case 2 — Same shape, single quotes

printf "The '_blank' word.\n" | cargo run --bin pampa --

Before:

Error: [Q-2-10] Closed Quote Without Matching Open Quote
   ╭─[ <stdin>:1:13 ]
   │
 1 │ The '_blank' word.
   │            ┬┬
   │            ╰─── This is the opening quote. If you need an apostrophe, escape it with a backslash.
   │             │
   │             ╰── A space is causing a quote mark to be interpreted as a quotation close.
───╯

After:

Error: [Q-2-5] Unclosed Underscore Emphasis
   ╭─[ <stdin>:1:12 ]
   │
 1 │ The '_blank' word.
   │      ┬     ┬
   │      ╰──────── This is the opening '_' mark.
   │            │
   │            ╰── Reached the end of the inline scope before finding a closing '_' for the emphasis.
───╯
ℹ If you want a literal underscore, escape it with `\_`.

The single-quote case was even more confusing on main — the diagnostic
talked about apostrophe interpretation while the actual problem was the
same unclosed _.

Case 3 — Unclosed emphasis inside single quotes (with whitespace flanks)

printf "'a * b'\n" | cargo run --bin pampa --

Before:

Error: [Q-2-7] Unclosed Single Quote
   ╭─[ <stdin>:1:8 ]
   │
 1 │ 'a * b'
   │       ┬┬
   │       ╰─── This is the opening quote. If you need an apostrophe, escape it with a backslash.
   │        │
   │        ╰── I reached the end of the block before finding a closing "'" for the quote.
───╯

After:

Error: [Q-2-12] Unclosed Star Emphasis
   ╭─[ <stdin>:1:7 ]
   │
 1 │ 'a * b'
   │    ┬  ┬
   │    ╰───── This is the opening '*' mark.
   │       │
   │       ╰── Reached the end of the inline scope before finding a closing '*' for the emphasis.
───╯
ℹ If you want a literal asterisk, escape it with `\*`.

main insisted the ' was unclosed even though both ' markers are
present and the actual problem is the orphan *. The new diagnostic
points at the * and uses "the inline scope" wording because the
indicator lands on the closing '.

Case 4 — Same emphasis at block level (no enclosing scope)

printf 'a * b\n' | cargo run --bin pampa --

Before:

Error: [Q-2-12] Unclosed Star Emphasis
   ╭─[ <stdin>:1:6 ]
   │
 1 │ a * b
   │ ┬    ┬
   │ ╰─────── This is the opening '*' mark.
   │      │
   │      ╰── I reached the end of the block before finding a closing '*' for the emphasis.
───╯

After:

Error: [Q-2-12] Unclosed Star Emphasis
   ╭─[ <stdin>:1:6 ]
   │
 1 │ a * b
   │   ┬  ┬
   │   ╰───── This is the opening '*' mark.
   │      │
   │      ╰── Reached the end of the block before finding a closing '*' for the emphasis.
───╯
ℹ If you want a literal asterisk, escape it with `\*`.

Two small improvements visible here even though the Q-code is unchanged:

1. The opening-mark indicator now points at the * (col 2) rather than
   the start of the line (col 0). On main, the opener anchor was off
   because the parser's leading-whitespace token was reported untrimmed;
   the new walker trims to the actual delimiter byte.
2. The wording stays "Reached the end of the block before…" — context-
   appropriate for a block-level failure with no enclosing inline scope.

Compare this against Case 3 above: the same Q-2-12 template, but the
wording switches between "the block" and "the inline scope" depending on
whether an outer closer anchors the indicator.

Case 5 — Unclosed inner emphasis inside paired outer emphasis

printf '*a _b c* trailing\n' | cargo run --bin pampa --

Before:

Error: [Q-2-12] Unclosed Star Emphasis
   ╭─[ <stdin>:1:18 ]
   │
 1 │ *a _b c* trailing
   │                  ┬
   │                  ╰── I reached the end of the block before finding a closing '*' for the emphasis.
───╯

After:

Error: [Q-2-5] Unclosed Underscore Emphasis
   ╭─[ <stdin>:1:8 ]
   │
 1 │ *a _b c* trailing
   │    ┬   ┬
   │    ╰────── This is the opening '_' mark.
   │        │
   │        ╰── Reached the end of the inline scope before finding a closing '_' for the emphasis.
───╯
ℹ If you want a literal underscore, escape it with `\_`.

main insisted the outer * was unclosed (and let the indicator float at
end-of-line, far from any source feature). The actual issue is the inner
_, with the outer *..* correctly closing at col 7. The new diagnostic
identifies the right marker and anchors at the closing *.

Case 6 — Truly unclosed single quote at block level

printf "'foo bar\n" | cargo run --bin pampa --

Before:

Error: [Q-2-7] Unclosed Single Quote
   ╭─[ <stdin>:1:9 ]
   │
 1 │ 'foo bar
   │ ┬       ┬
   │ ╰────────── This is the opening quote. If you need an apostrophe, escape it with a backslash.
   │         │
   │         ╰── I reached the end of the block before finding a closing "'" for the quote.
───╯

After:

Error: [Q-2-9] Unclosed Single Quote
   ╭─[ <stdin>:1:9 ]
   │
 1 │ 'foo bar
   │ ┬       ┬
   │ ╰────────── This is the opening single quote.
   │         │
   │         ╰── Reached the end of the block before finding a closing single quote.
───╯
ℹ If you meant a literal apostrophe, escape it with `\'`.

The behavior is essentially unchanged here — both branches correctly
identify the unclosed ' at block level. The Q-code has shifted from
Q-2-7 to Q-2-9 (a new code introduced earlier in the branch to
distinguish whitespace-prefixed openers from the apostrophe-as-text-close
interpretation that qmd-syntax-helper relies on), but the diagnostic
content is equivalent.

The bug

The misclassification is a consequence of how tree-sitter's LR state
generator minimises distinct states with identical outgoing transitions.
Multiple input shapes that semantically fail for different reasons
collapse onto the same (LR state, lookahead symbol) parse-error pair,
and the Merr-style error table can only see that pair. For example, both
of these reach (state=704, sym=_whitespace) at error time:

Input	Real failure	Wanted
_a' b._	apostrophe-as-text-close collides with _..._	Q-2-10 "Closed Quote Without Matching Open"
The '_blank' word.	unclosed _ inside paired '..'	Q-2-5 "Unclosed Underscore Emphasis"

The same shape repeats across all six "Unclosed X" inline scopes (single
quote, double quote, single/double emphasis with * or _).

Approach: scope-stack walker plus context-aware dispatch

The core idea is to give the dispatcher a third piece of context beyond
the LR state and lookahead: which inline scope was open at error time.
This is recovered by walking the parser's token log.

outer_scope.rs walker

crates/quarto-parse-errors/src/outer_scope.rs (new, 740 LOC) provides
one internal scope_walk helper plus three public readers:

• compute_outer_scope — returns the innermost open inline scope at the
  error position (the top of the stack).
• find_outermost_close — returns the position of the would-be closer
  for the outermost open scope (the bottom of the stack), or None if
  no candidate closer exists in the remaining tokens.
• find_innermost_open_position — returns the position of the innermost
  open scope's opening delimiter.

The walker combines the parser's all_tokens and consumed_tokens logs,
sorts them by source position, and iterates applying the same push /
pop-if-top / block-boundary-clear rules the external scanner uses. The
result is the scope-stack snapshot at the moment the parser failed.

Recognised scope kinds: single_quote, double_quote, emphasis
delimiters (* and _, distinguished by reading the actual byte from
the input because emphasis tokens can include leading whitespace), and
strong emphasis delimiters.

Corpus-driven OuterScope → Q-code table

Each "Unclosed X" corpus file carries a top-level annotation:

{
  "code": "Q-2-5",
  "title": "Unclosed Underscore Emphasis",
  "unclosedScope": "emph_underscore",
  ...
}

build_error_table.ts collects these into a generated artifact
_autogen-scope-owners.json (6 entries), and a sibling proc-macro
include_scope_owner_table! embeds it at compile time.
OuterScope::to_qcode reads from this generated table rather than a
hardcoded match, so renaming a Q-code in the corpus propagates
automatically — the same trust model the Merr table itself uses.

Dispatch in error_generation.rs

For each parse-error state, error_diagnostic_from_parse_state:

1. Computes the outer scope and the would-be closer of the outermost
   enclosing scope.
2. Decides whether to route the diagnostic through outer-scope dispatch
   directly or through the Merr (state, sym) lookup:
   • Emphasis-flavored or double_quote innermost scope →
     outer-scope dispatch: build the diagnostic from the scope's owning
     Q-code template and skip the Merr table entirely. This handles
     The "_blank" word.-style cases as well as arbitrary nesting
     depth (3+ levels) without needing per-shape corpus entries.
   • single_quote innermost scope → fall through to Merr by
     default, because the apostrophe-as-close heuristic (a '
     preceded by a letter is text, not an opener) means the walker's
     view can diverge from the parser's view here. Pre-existing
     Q-2-7 / Q-2-10 entries handle the parser-interpretation
     overrides correctly. When Merr's match does not carry one of
     those apostrophe-heuristic codes — i.e. the match is a
     state-collision false positive from some other input that
     happened to reach the same (state, sym) — fall back to the
     synthetic Q-2-9 path.
   • No outer scope → Merr lookup as usual.
3. Anchors the primary indicator at the closing delimiter of the
   outermost open scope (via find_outermost_close) when one exists;
   otherwise falls back to the parser-failure position.
4. Substitutes the {anchor} placeholder in the corpus message with
   "the inline scope" or "the block" depending on which anchor branch
   fired in step 3.

The asymmetric handling of single-quote vs. the other scopes is
principled, not a hack: single-quote is the one scope kind in qmd where
the parser applies a heuristic that diverges from a naive push/pop
model of the source characters. Every other inline delimiter has
unambiguous open/close semantics, so the walker's stack and the parser's
interpretation agree.

Q-2-9 vs. Q-2-10: the apostrophe-heuristic split

Single-quote is the one scope kind that needs two distinct
"unclosed-X" diagnostics rather than one, because qmd's apostrophe
heuristic gives ' two parser-side interpretations. The walker pushes
SingleQuote on every ' token it sees; the parser decides per-'
whether the character is an opener or a close-apostrophe based on
what precedes it.

• Q-2-9 "Unclosed Single Quote" is for shapes where the ' is
  a real opener — typically whitespace-prefixed — and the user forgot
  the matching close. Example: 'foo bar (Case 6 above), or
  *a 'b c* (an unclosed inner quote inside paired emphasis). The
  walker's stack matches the parser's view here, so the dispatcher
  hits the synthetic Q-2-9 path through OuterScope::to_qcode.

• Q-2-10 "Closed Quote Without Matching Open Quote" is for shapes
  where the ' follows a letter, so the parser treats it as text
  (an apostrophe-as-close), while some surrounding scope is what
  actually failed. Example: _a' b._ — the parser's view is
  "stray apostrophe close inside emphasis"; the user's actual
  problem is the unmatched _. (qmd-syntax-helper's auto-fix
  pipeline depends on Q-2-10 firing for these shapes so it can
  suggest escaping the apostrophe.)

The two interpretations frequently land on the same (LR state, lookahead) parse-error bucket, so the table-only approach can't
distinguish them. The dispatch rule that resolves this is in the
single-quote arm of the gate:

When the innermost open scope is SingleQuote, look at the Merr
entries for the current (state, sym). If any entry carries
Q-2-7 or Q-2-10, take it — those are the parser-interpretation
overrides authored deliberately into the corpus. If neither code
is present, the matched entry is a state-collision false positive
from some unrelated shape that happened to reach the same bucket;
fall through to the synthetic Q-2-9 path that the walker's view
suggests.

In effect, Q-2-7 and Q-2-10 act as explicit overrides for
parser-side reinterpretations of ', and Q-2-9 is the default
for everything else with a SingleQuote on the stack. The dispatch
gate codes this check inline (matches!(e.error_info.code, Some("Q-2-7") | Some("Q-2-10"))) rather than via another corpus annotation; with
only two override codes and no path to growth (apostrophe is the only
heuristic of its kind in qmd), the hardcoded set is more readable than
a generated table would be. If a future inline delimiter grows a
similar parser-side heuristic, the same pattern can be lifted into a
corpus annotation at that point.

Unrelated fixes folded in

Scanner ::: peek gate (in commit a01e3bc6)

crates/tree-sitter-qmd/tree-sitter-markdown/src/scanner.c had a
regression where the pipe-table-line-ending detection's :::
lookahead peek ran in any context, not just when
PIPE_TABLE_LINE_ENDING was a valid symbol. The peek calls advance()
to step past the colons; in non-pipe-table contexts this corrupted
the lexer's position. Subsequent SOFT_LINE_ENDING emissions then
mark_end-ed at the wrong byte, breaking fenced divs whose body
contained an inline-bearing block followed by :::{attrs} on the next
line.

Fix: gate the peek on valid_symbols[PIPE_TABLE_LINE_ENDING]. 19-line
change with a long comment explaining the failure mode in detail.

GLR speculative-error filter (in commit a01e3bc6)

Already described above. When had_errors() is true but the final
tree has no ERROR nodes, treat the parse as successful — the errors
came from dead GLR speculation branches.

What this branch changes structurally

New files

• crates/quarto-parse-errors/src/outer_scope.rs — scope-stack walker
  and OuterScope enum.
• crates/pampa/resources/error-corpus/_autogen-scope-owners.json —
  generated OuterScope → Q-code mapping.
• crates/pampa/tests/test_emphasis_in_quote.rs — 61-test regression
  suite covering all the bug shapes, control cases, and deep nesting.

Modified APIs

• produce_diagnostic_messages — new scope_owners: &[ScopeOwnerEntry] parameter. Pampa's wrapper supplies
  get_scope_owner_table().
• OuterScope::to_qcode — now takes a &[ScopeOwnerEntry] parameter.
• The include_scope_owner_table! proc-macro is a new sibling to
  include_error_table!, sharing the same input syntax.

Unchanged APIs

• lookup_error_message, lookup_error_entry, ErrorTableEntry — all
  back to their pre-branch signatures (no outer_scope parameter or
  field).
• produce_error_message_json — back to its pre-branch signature (no
  input_bytes parameter).

Corpus contract

Each "Unclosed X" Q-*.json file now declares:

{
  "code": "Q-2-X",
  "title": "Unclosed X",
  "unclosedScope": "<scope-name>",
  "message": "Reached the end of {anchor} before finding ...",
  ...
}

Extending to other inline scopes

The walker today recognises only emphasis and quote scopes. The same
bug class exists for every other inline kind in qmd: span [..],
image ![..], inline footnote ^[..], code span `..`, inline
math $..$, superscript ^..^, subscript ~..~, strikeout
~~..~~, editorial markers ({+..+}, {-..-}, {>..<}, {!..!},
{=..=}), and inline attribute specifiers {..}.

A quick probe shows these are still misclassified on both main and
this branch:

printf 'The "`code foo" word.\n' | cargo run --bin pampa --

Error: [Q-2-11] Unclosed Double Quote
   ╭─[ <stdin>:1:6 ]
   │
 1 │ The "`code foo" word.
   │     ┬┬
   │     ╰─── This is the opening quote mark.
   │      │
   │      ╰── Reached the end of the block before finding a closing '"' for the quote.
───╯

The diagnostic should say [Q-2-24] Unclosed Code Span and point at the
backtick. The walker doesn't track code-span tokens, so the dispatcher
sees outer_scope = double_quote and emits the (incorrect) Q-2-11
template.

This is now a mechanical fix, not a design problem. Adding a new
inline scope kind is exactly three steps:

1. Walker recognition. Add a new variant on OuterScope (e.g.
   CodeSpan) and a matching arm in scope_for_token mapping the
   tree-sitter token kind (e.g. code_span_delimiter) to that
   variant. This is the only parser-bound piece — it requires knowing
   what tokens the grammar emits, which is inherently in code.
2. Corpus annotation. Add "unclosedScope": "code_span" to the
   corresponding Q-*.json corpus file (e.g. Q-2-24.json for
   unclosed code span). Regenerate via
   ./scripts/build_error_table.ts. The build script automatically
   pulls the annotation into _autogen-scope-owners.json, and
   OuterScope::to_qcode will find the new mapping at compile time.
3. Dispatch gate (if needed). Check whether the new scope kind has
   any parser-side heuristic that diverges from the walker's view (the
   way single-quote does with its apostrophe-as-text-close rule). If
   yes — add an arm to the dispatch gate in error_generation.rs that
   routes that scope through Merr. If no — let it use the default
   outer-scope dispatch like emphasis and double-quote already do.
   Most inline scopes have unambiguous open/close semantics, so this
   step is a no-op for them.

No new corpus entries per (LR state, lookahead) bucket. No new schema
fields. No combinatorial expansion across N outer × M inner pairs. The
infrastructure built in Phases 2–4 of this branch was specifically
designed so that each new inline scope adds constant work, not
quadratic.

Possible parser-side heuristics worth checking for the remaining inline
kinds, in case they need step 3:

• Span vs. attribute specifier {..} — Pandoc's syntax overlaps;
  pampa may treat { after a span specially. Worth checking the
  scanner for context-sensitive cases.
• Math $..$ — $ followed by a digit-or-letter is math; $ in
  prose contexts is just a dollar sign. Like the apostrophe heuristic,
  this is a parser-level distinction the walker would need to mirror
  if it pushed naively on every $.
• Superscript ^..^ vs footnote prefix ^[..] — distinguishable
  by lookahead, but the walker would need to consult the next token
  before deciding which variant of ^ to push.
• Strikeout ~~..~~ vs subscript ~..~ — distinguishable by
  token kind in tree-sitter, so the walker handles them with separate
  scope variants.

Each of these is a self-contained follow-up issue. The framework
established here scales to all of them without architectural change.

Coverage

Test counts

• test_emphasis_in_quote.rs: 61 tests covering all six "Unclosed X"
  shapes across single/double quote, emphasis-star/underscore, and
  strong-star/underscore outer wrappers, plus controls for
  apostrophe-in-emphasis, deep nesting, multi-paragraph regression
  shapes, and the new anchor-wording disambiguation.
• Workspace: 9,072 tests pass.

CI parity

cargo xtask verify --skip-hub-build --skip-hub-tests --skip-trace-viewer-build --skip-trace-viewer-tests passes: lint,
fmt, build with -D warnings, tree-sitter tests, workspace nextest.

Known limitations

These cases are documented here rather than addressed in this PR
because each requires independent design work larger than the scope of
this branch.

Multiple unclosed inlines in a single block

printf 'a *b _c\n' | cargo run --bin pampa --

Before and after both produce only one diagnostic (for the innermost
_), even though both * and _ are unclosed:

Error: [Q-2-5] Unclosed Underscore Emphasis
   ╭─[ <stdin>:1:8 ]
   │
 1 │ a *b _c
   │      ┬ ┬
   │      ╰──── This is the opening '_' mark.
   │        │
   │        ╰── Reached the end of the block before finding a closing '_' for the emphasis.
───╯

The walker correctly sees both on its stack, but the dispatcher emits
one diagnostic per parser error state and currently picks the
innermost frame. Fixable in the dispatch layer (~20 LOC): when
find_outermost_close returns None, emit one diagnostic per stack
frame outermost-first.

Errors in subsequent paragraphs after the first failure

printf 'a *b\n\nc _d\n' | cargo run --bin pampa --

Only the first paragraph's error is reported; the second paragraph's
unclosed _ is silent. The verbose trace shows tree-sitter hits
detect_error once, then enters recover_to_previous and
skip_token-s through the rest of the document without re-entering a
productive state — so no second parse-error event ever fires.

This is a parser-level limitation, not a dispatch-layer one. Possible
approaches for a follow-up:

• Walk ERROR nodes in the final tree to find blocks with unreported
  failures, then re-run the scope-walker on each.
• Re-invoke the parser per block on failure, syncing at block
  boundaries.
• Customise tree-sitter recovery so the parser resyncs at block-start
  instead of skipping to EOF.

The first option is the most plausible without touching tree-sitter
internals and would reuse the existing walker.

[gordonwoodhull]

😮

[cscheid]

👀

Approach: scope-stack walker plus context-aware dispatch

The core idea is to give the dispatcher a third piece of context beyond
the LR state and lookahead: which inline scope was open at error time.

This is a little terrifying. I... I don't know if I want to merge this before understanding this very thoroughly!

[cscheid]

The "_blank" word

Yeah, this is all a bit cursed (yay Markdown). Consider the following:

The "_blank "word" oh no_".

Should this parse correctly? I can make arguments in both directions. Commonmark doesn't appear to say anything about which way it would go, but it does have opinions about nested links, namely that they're not allowed (https://spec.commonmark.org/0.31.2/#links). So one could argue that the spirit of Commonmark would require us to disallow nested quotations. This in principle can be solved by created a (much) more complex version of the grammar that restricts all inlines inside of quotations to be non-quotation inlines.

But there's an argument to be made that nested quotations should be allowed, actually. For example,

"'this' is defensible"

And if that's an acceptable Markdown document, why shouldn't this be?

"'"This" is' pushing it."

And from there, the only reason we don't support truly nested quotes is typographical. But _ introduces emphasis, and so why not allow our original?

The "_blank "word" oh no_".

I note in passing that this example parses successfully in pampa:

$ cat c2.qmd
The "_blank "word" oh no_".%
$ pampa -i c2.qmd -t native
Warning [Q-7-1]: Missing Newline at End of File
File `c2.qmd` does not end with a newline

[ Para [Str "The", Space, Quoted DoubleQuote [Emph [Str "blank", Space, Quoted DoubleQuote [Str "word"], Space, Str "oh", Space, Str "no"]], Str "."] ]

[cscheid]

More generally:

How hard could that possibly be to fix?

:lolsob: I get it, I think you're going through the same process I went when I learned of this parsing technique: "oh my god it solves everything" three weeks later "oh no, this is kind of stiflingly limited actually".

In general, I think your proposal is a version of something I thought of: we could benefit from using a stack of context "unreduced shifts" in a direct generalization of the Jeffery technique. At the point where we read the token after blank in "_blank, we know we're inside a double quotation, and inside an underscore emphasis. In full generality, we could have a system that matches on regular expressions of suffixes of those inline scopes.

But, to be quite frank, I don't want to spend time trying to study this. I was an academic for a decade. I just want to build tools for people to use :) I genuinely think the risks outweigh the benefits here. We're already playing with fire by doing this trick inside tree-sitter and needing to get around its error correction. I don't want to compound that risk.

[rundel]

Totally fair - I in no way can claim to understand a fraction of the code generated, this was the result of many iterations where the proposed solution either did not work or got way more invasive in terms of the necessary changes in the corpuses or scanner.c or even worse.

This was the closest I got to something that felt at least like a semi-plausible solution.

[rundel]

In general, I think your proposal is a version of something I thought of: we could benefit from using a stack of context "unreduced shifts" in a direct generalization of the Jeffery technique. At the point where we read the token after blank in "_blank, we know we're inside a double quotation, and inside an underscore emphasis. In full generality, we could have a system that matches on regular expressions of suffixes of those inline scopes.

This was the direction I was originally interested in taking - claude was able to modify scanner.c such that it successfully built a context stack that kept track of open and closed inlines via pushing and popping but I wasn't able to get that passed into tree-sitter / the jeffery technique in such a way that it could be surfaced as a useful error.