Add raw string literals (r'…') ahead of making triple quotes non-raw - #47
Merged
Merged
Conversation
Triple-quoted literals were fully raw while single-delimiter strings processed a seven-character escape whitelist. That split made the escaping regime depend on how many quote characters were counted: `\'` is correct in `'…'` and ships verbatim in `'''…'''`, with nothing else distinguishing them. LLM-authored Forthic does not survive that. Over a 20,000-program production week, 93% of every backslash occurrence inside a triple-quoted literal is a `\'` or `\"` the model wrote expecting it to be an escape, and the corruption reaches users. Replaying the same week through this change drops apostrophe corruption inside `'''` from 22.97% to 14.91%, and double-quote corruption inside `"""` from 49.02% to 1.96%. No language treats `'''` as raw. Python and Groovy are the only languages that have it and both process escapes; the raw-triple-quote convention belongs to `"""` in C-family languages where `'` is a character literal and `'''` does not exist. Forthic paired Python's spelling with Kotlin's semantics. The whitelist is what keeps this safe rather than a wholesale move to escaped strings: only \n \t \r \0 \\ \" \' are interpreted, so `\d`, `\w` and `\U` still stay literal and regexes and Windows paths remain writable. Across the committed corpus (2,887 literals) exactly two sites change meaning, both in a gdrive example that ships a literal `\n` into a Google Doc where a newline was plainly intended. Escapes resolve before delimiter detection, so an escaped quote is content and can never close the literal. Unescaped tripling (`'''today'''s plan'''`) still closes early — this change moves the silent classes, not the loud one. Three tests asserting the previous raw behaviour are updated; the regex and path cases they sat beside are untouched and still pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011UUMTq8x3R5KN1pxKJa4ua
liamfd
force-pushed
the
liam/big-7145-triple-quote-escapes
branch
2 times, most recently
from
August 14, 2026 15:36
b1c7a69 to
b5210c6
Compare
Forthic strings carry two jobs that one delimiter cannot serve. Regular strings
interpret a seven-escape whitelist, so an author writing 'today\'s' gets an
apostrophe. Triple-quoted strings carry data, where a backslash has to survive
untouched: JSON payloads and embedded Forthic source parse their own escapes,
and any escape the tokenizer consumes is one the inner layer never sees.
Serving the first job at triple width breaks the second. '''{"a": "x\ny"}'''
JSON> fails on a raw control character, '''{"p": "C:\\Users"}''' JSON> on an
invalid \U, and """ 'a\\nb' """ RUN processes the escape twice. Leaving triple
quotes raw to protect the second leaves the first broken — an LLM writes
'''today\'s plan''' because that is correct one delimiter width narrower, and
the backslash reaches users. Both are paths docs/forthic-prompt.md documents.
An explicit raw form separates the jobs. `r` glued to any opening delimiter
turns escape processing off for that literal: r'…', r"…", r'''…''', r"""…""".
This is a transitional release, not the destination. Triple-quoted strings are
still raw, so the triple-width r forms are aliases today. They exist so data
literals can move before the next release makes '''…''' and """…""" interpret
the whitelist, at which point a bare triple-quoted string stops being a safe
place for a backslash. This is the only window in which both spellings work.
The tokenizer gains is_raw_string_start() and one branch in
transition_from_START, mirroring the existing <<'''…''' marked-redirect
lookahead, plus a raw flag on transition_from_GATHER_STRING that skips its
escape branch. The triple-quote gather does no escape processing, so the
triple-width forms reuse it unchanged. The prompt routes backslash-carrying
strings to the r forms and warns that the bare form will not keep a backslash.
`r` is the strongest prior available (Python, Rust, Nim, D, Scala) and Forthic
already borrowed Python's ''' spelling. Backticks were ruled out despite Go and
JS: generated Forthic is routinely wrapped in markdown fences, where
triple-backtick collides. The prefix is lowercase only and does not compose with
the redirect marker. Unlike Python, a raw Forthic string may end in a backslash
— r'C:\' is C:\ — though the same rule means it cannot contain its own
delimiter.
One narrow break rather than a pure addition: words are gathered without
breaking on quote characters, so a token spelled r + quote (r'don't') used to
lex as a single word and now opens a raw string. No standard-library word is
spelled that way, and definition names cannot contain quotes, so no definable
word is affected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N1Xfa8zk25MrXCZcUEPWY7
liamfd
force-pushed
the
liam/big-7145-triple-quote-escapes
branch
from
August 14, 2026 15:42
b5210c6 to
a9fb3f8
Compare
liamfd
marked this pull request as ready for review
August 14, 2026 15:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Transitional release. Adds
r-prefixed raw string literals at every delimiter width —r'…',r"…",r'''…''',r"""…"""— so that backslash-carrying data has somewhere to live before the next release makes bare'''…'''and"""…"""interpret the escape whitelist. After that, a bare triple-quoted string stops being a safe place for a backslash. This is the only window in which both spellings work.Why. Triple quotes are carrying two jobs that one delimiter can't serve. They need to interpret escapes, so
'''today\'s plan'''stops shipping its backslash to users. They also need to stay raw, becauseJSONandRUNparse their own escapes and any escape the tokenizer eats is one the inner layer never sees:Both sit on paths
docs/forthic-prompt.mddocuments. Splitting the jobs across two spellings serves both; narrowing the whitelist would serve neither fully.Structure.
is_raw_string_start()plus one branch intransition_from_START, mirroring the existing<<'''…'''marked-redirect lookahead, and arawflag ontransition_from_GATHER_STRING. The triple-quote gather does no escape processing, so the triple-width forms reuse it unchanged and are aliases today. The prompt presentsras an escape hatch for content whose escapes something else has to read — not as a blanket "contains a backslash" rule — and warns that the bare triple form won't keep a backslash.Compatibility. One narrow break, not a pure addition: words are gathered without breaking on quote characters, so a token spelled
r+ quote (r'don't') used to lex as a single word and now opens a raw string. No standard-library word is spelled that way, and definition names can't contain quotes, so no definable word is affected. Hence the patch release.Design.
ris the strongest prior available (Python, Rust, Nim, D, Scala) and Forthic already borrowed Python's'''. Backticks were ruled out despite Go and JS — generated Forthic is routinely wrapped in markdown fences. Lowercase only; no composition with the redirect marker. Unlike Python, a raw Forthic string may end in a backslash (r'C:\'→C:\), though the same rule means it can't contain its own delimiter.Testing. 1011 passing, plus build and smoke (including the self-verifying prompt-examples check). Covers the JSON/
RUN/path idioms end to end, greedy quote-runs with the prefix, mid-word collision safety, declined redirect composition, empty and unterminated raw strings, and anr-string streamed across every chunk boundary. Disabling the triple-width dispatch fails 9 tests.Deferred to the breaking release, where the escape flip makes them testable: a streaming chunk boundary landing on a backslash collapses the cumulative value instead of growing it, and
StringRedirectRouter.feed()diffs by length alone — so the sink and the stack would silently disagree.🤖 Generated with Claude Code
https://claude.ai/code/session_01N1Xfa8zk25MrXCZcUEPWY7