Make triple-quoted strings interpret the escape whitelist - #48
Merged
rjose merged 8 commits intoAug 14, 2026
Conversation
Triple-quoted literals were fully raw while single-delimiter strings processed a seven-character escape whitelist. That split made the escaping regime depend on how many quote characters were counted: `\'` is correct in `'…'` and ships verbatim in `'''…'''`, with nothing else distinguishing them. LLM-authored Forthic does not survive that. Over a 20,000-program production week, 93% of every backslash occurrence inside a triple-quoted literal is a `\'` or `\"` the model wrote expecting it to be an escape, and the corruption reaches users. Replaying the same week through this change drops apostrophe corruption inside `'''` from 22.97% to 14.91%, and double-quote corruption inside `"""` from 49.02% to 1.96%. No language treats `'''` as raw. Python and Groovy are the only languages that have it and both process escapes; the raw-triple-quote convention belongs to `"""` in C-family languages where `'` is a character literal and `'''` does not exist. Forthic paired Python's spelling with Kotlin's semantics. The whitelist is what keeps this safe rather than a wholesale move to escaped strings: only \n \t \r \0 \\ \" \' are interpreted, so `\d`, `\w` and `\U` still stay literal and regexes and Windows paths remain writable. Across the committed corpus (2,887 literals) exactly two sites change meaning, both in a gdrive example that ships a literal `\n` into a Google Doc where a newline was plainly intended. Escapes resolve before delimiter detection, so an escaped quote is content and can never close the literal. Unescaped tripling (`'''today'''s plan'''`) still closes early — this change moves the silent classes, not the loud one. Three tests asserting the previous raw behaviour are updated; the regex and path cases they sat beside are untouched and still pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UUMTq8x3R5KN1pxKJa4ua
Forthic strings carry two jobs that one delimiter cannot serve. Regular strings
interpret a seven-escape whitelist, so an author writing 'today\'s' gets an
apostrophe. Triple-quoted strings carry data, where a backslash has to survive
untouched: JSON payloads and embedded Forthic source parse their own escapes,
and any escape the tokenizer consumes is one the inner layer never sees.
Serving the first job at triple width breaks the second. '''{"a": "x\ny"}'''
JSON> fails on a raw control character, '''{"p": "C:\\Users"}''' JSON> on an
invalid \U, and """ 'a\\nb' """ RUN processes the escape twice. Leaving triple
quotes raw to protect the second leaves the first broken — an LLM writes
'''today\'s plan''' because that is correct one delimiter width narrower, and
the backslash reaches users. Both are paths docs/forthic-prompt.md documents.
An explicit raw form separates the jobs. `r` glued to any opening delimiter
turns escape processing off for that literal: r'…', r"…", r'''…''', r"""…""".
This is a transitional release, not the destination. Triple-quoted strings are
still raw, so the triple-width r forms are aliases today. They exist so data
literals can move before the next release makes '''…''' and """…""" interpret
the whitelist, at which point a bare triple-quoted string stops being a safe
place for a backslash. This is the only window in which both spellings work.
The tokenizer gains is_raw_string_start() and one branch in
transition_from_START, mirroring the existing <<'''…''' marked-redirect
lookahead, plus a raw flag on transition_from_GATHER_STRING that skips its
escape branch. The triple-quote gather does no escape processing, so the
triple-width forms reuse it unchanged. The prompt routes backslash-carrying
strings to the r forms and warns that the bare form will not keep a backslash.
`r` is the strongest prior available (Python, Rust, Nim, D, Scala) and Forthic
already borrowed Python's ''' spelling. Backticks were ruled out despite Go and
JS: generated Forthic is routinely wrapped in markdown fences, where
triple-backtick collides. The prefix is lowercase only and does not compose with
the redirect marker. Unlike Python, a raw Forthic string may end in a backslash
— r'C:\' is C:\ — though the same rule means it cannot contain its own
delimiter.
One narrow break rather than a pure addition: words are gathered without
breaking on quote characters, so a token spelled r + quote (r'don't') used to
lex as a single word and now opens a raw string. No standard-library word is
spelled that way, and definition names cannot contain quotes, so no definable
word is affected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N1Xfa8zk25MrXCZcUEPWY7
The escaping regime depended on how many quote characters were counted. `\'` was correct in '...' and shipped verbatim in '''...''', with nothing else in the source to tell them apart. LLM-authored Forthic does not survive that: over a 20,000-program production week, 93% of every backslash inside a triple-quoted literal is a \' or \" the model wrote expecting an escape, and the backslash reaches users. 0.16.2 could not simply fix it, because triple quotes were carrying a second job — they were the raw, data-carrying form, and JSON> and RUN parse their own escapes. That release split the jobs by adding an explicit raw prefix and pre-announced this one. This is the second half: '''...''' and """...""" now interpret \n \t \r \0 \\ \" \', and r'''...''' is the only raw spelling at triple width. The whitelist is what keeps this safe rather than a wholesale move to escaped strings. \d, \w, \U and \. stay literal, so regexes and Windows paths are still writable without doubling. Escapes resolve before delimiter detection, so an escaped quote is content and can never close the literal — '''a \'\'\' b''' is a ''' b. Unescaped tripling still closes early: this moves the silent failure classes, not the loud one. One new loud break: a triple-quoted literal ending in a backslash no longer parses. '''C:\''' consumes \' as an escaped quote, leaving '' that cannot close. Write r'''C:\'''. Fixes a streaming corruption the flip would otherwise arm. streamingRun re-tokenizes the whole cumulative buffer each chunk and reports the open string's content-so-far to its redirect sink, which diffs by length. A backslash landing as the last character of a chunk was reported as content, then re-read as an escape next chunk — so the value could shrink, or stay the same length while changing, and the sink received a\b where the stack held a real newline. get_string_value() now holds back a trailing backslash while a triple-quoted string is open, exactly as it already held back an unconfirmed closing quote. Both halves land together; there is no commit where the flip exists without the hold-back. The test that covers it compares the sink's concatenated deltas against the string actually left on the stack, with no hardcoded expected value, because pinning the text would let a future semantics change mask a divergence. It was confirmed to fail without the hold-back and pass with it. Note that a doubled backslash still yields a growing prefix at every cut, so it is kept as a control rather than evidence. The prompt's Strings section is rewritten around what models were guessing at. It now states one rule for all four widths, says outright that an escape resolves before the closing delimiter is looked for, shows \' inside '''...''' and \" inside """...""", names the unescaped-tripling failure, and puts "change the delimiter instead" ahead of escaping as the better habit. Eleven of its examples are written in the `code` -> `expected` form the smoke test executes, up from three, so the semantics that were being guessed at are now machine-checked. Prompt docs are generated; the generator was edited. export_state captures definition source verbatim and import_state re-tokenizes it, so a 0.16.x state blob whose definitions hold bare triple literals with whitelisted escapes changes meaning on import. Round-trip tests pin which spelling survives the boundary, and the changelog says so. Not included, and tracked separately as urgent: unescape_string() rewrites < and > across the whole cumulative buffer before tokenizing, so a chunk boundary inside one of those entities makes the reported value mutate rather than extend — the sink receives abc&l while the stack holds abc<X, at equal length, so nothing corrects it. That predates this change and is not caused by it. The router-side prefix-invariant guard belongs with that fix rather than here, since it would otherwise throw on a legitimate stream. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012tWcQ3X7VpqUerMEgnZj8o
liamfd
marked this pull request as draft
August 14, 2026 17:06
The CHANGELOG listed '''C:\Users\tmp''' among the literals left unchanged by
the escape flip. \t is in the whitelist, so that literal now yields
C:\Users + TAB + mp — the guidance told readers to skip exactly the paths
that break. Replace the example with one that survives, spell out the
overlap, and say what to grep for.
Pin the semantics with a test: the mixed case was present in the redirect
payload table but only checked for sink/stack consistency, so nothing caught
the false claim. Its comment described it as non-whitelist too.
Also trim the comments the escape work added. The hold-back rule was stated
three times (field, docblock, setter) — keep the full version on
get_string_value() and point the other two at it. Drop the paragraph
justifying one loop over two passes down to the clause that carries it, and
the non-escape branch comment that repeated the ESCAPE_MAP docblock verbatim.
Reword release-relative framing in tests ("the flip", "the 0.16.2 window is
closed") to state the rule instead, which is what still reads correctly once
0.17.0 is several releases back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TPasz3i77ev8qk1RCSsgey
liamfd
marked this pull request as ready for review
August 14, 2026 20:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #47 — merge that first. This targets
mainonly because the base branch lives on a fork, so the diff currently shows both commits. Once #47 merges, it reduces on its own to the single commit here (Make triple-quoted strings interpret the escape whitelist) — that is the one to review.Why
Forthic's escaping regime depended on how many quote characters you counted.
\'was correct in'…'and shipped verbatim in'''…''', with nothing in the source to tell the two apart.LLM-authored Forthic does not survive that. Over a 20,000-program production week, 93% of every backslash inside a triple-quoted literal is a
\'or\"the model wrote expecting an escape — and the backslash reached users.0.16.2 (#47) couldn't just fix it: triple quotes were carrying a second job as the raw, data-carrying form, and
JSON>/RUNparse their own escapes. That release split the jobs by adding an explicit raw prefix and pre-announced this one. This is the second half.What
Triple-quoted strings now escape exactly the way single-quoted strings already do. Nothing new is introduced —
'…'and"…"have always interpreted\n \t \r \0 \\ \" \', and this removes the special case where'''…'''and"""…"""didn't. One rule at every delimiter width, andr'''…'''to opt out.That also means the safety properties are the existing ones, not new ones. The whitelist is what keeps this from being a wholesale move to escaped strings:
\d,\w,\Uand\.stay literal at triple width just as they already do at single width, so regexes and Windows paths remain writable without doubling. Escapes resolve before delimiter detection, so an escaped quote is content and can never close the literal. Unescaped tripling still closes early — this moves the silent failure classes, not the loud one.One new loud break: a triple-quoted literal ending in a backslash no longer parses.
'''C:\'''consumes\'as an escaped quote, leaving''that cannot close. Writer'''C:\'''.Fixes a streaming corruption the flip would otherwise arm. A backslash landing as the last character of a streamed chunk was reported to a redirect sink as content, then re-read as an escape on the next chunk — so the sink received
a\bwhere the stack held a real newline. Both halves land in the same commit; there is no commit where the flip exists without the fix.The prompt docs are rewritten around what models were guessing at. Production experience with the 0.16.1 prompt was that models invent escaping semantics when the docs don't state them. It now gives one rule for all four widths, says outright that an escape resolves before the closing delimiter is looked for, shows
\'inside'''…'''and\"inside"""…""", names the unescaped-tripling trap, and puts "change the delimiter instead" ahead of escaping. Eleven of its examples are now written in the form the smoke test executes, up from three — so the semantics that were being guessed at are machine-checked rather than asserted.Migration
Mechanical: prefix any triple-quoted literal containing a backslash with
r. Literals without one are unaffected. Traps called out in the changelog: JSON payloads, embedded Forthic throughRUN/MAP/FILTER/WHEN, literals ending in a backslash, and redirect literals (which have no raw spelling — double the backslash).export_statestores definition source verbatim andimport_statere-tokenizes it, so a 0.16.x state blob holding bare triple literals with escapes changes meaning on import. Round-trip tests pin which spelling survives.Verification
JSON>andRUNidioms that motivated the two-release split were driven end to end and are untouched.Deliberately not here
unescape_string()rewrites</>across the whole cumulative buffer before tokenizing, so a chunk boundary inside one of those entities makes the reported value mutate rather than extend — the sink getsabc&lwhile the stack holdsabc<X, at equal length, so nothing corrects it. That predates this change and is not caused by it. Tracked separately as urgent. The router-side prefix-invariant guard belongs with that fix rather than here, since it would otherwise throw on a legitimate stream.🤖 Generated with Claude Code
https://claude.ai/code/session_012tWcQ3X7VpqUerMEgnZj8o