Skip to content

Add raw string literals (r'…') ahead of making triple quotes non-raw - #47

Merged
rjose merged 2 commits into
forthix:mainfrom
liamfd:liam/big-7145-triple-quote-escapes
Aug 14, 2026
Merged

Add raw string literals (r'…') ahead of making triple quotes non-raw#47
rjose merged 2 commits into
forthix:mainfrom
liamfd:liam/big-7145-triple-quote-escapes

Conversation

@liamfd

@liamfd liamfd commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Transitional release. Adds r-prefixed raw string literals at every delimiter width — r'…', r"…", r'''…''', r"""…""" — so that backslash-carrying data has somewhere to live before the next release makes bare '''…''' and """…""" interpret the escape whitelist. After that, a bare triple-quoted string stops being a safe place for a backslash. This is the only window in which both spellings work.

Why. Triple quotes are carrying two jobs that one delimiter can't serve. They need to interpret escapes, so '''today\'s plan''' stops shipping its backslash to users. They also need to stay raw, because JSON and RUN parse their own escapes and any escape the tokenizer eats is one the inner layer never sees:

'''{"a": "x\ny"}''' JSON>   ->  Bad control character in string literal
'''{"p": "C:\\Users"}''' JSON>  ->  Bad escaped character in JSON
""" 'a\\nb' """ RUN         ->  escapes processed twice

Both sit on paths docs/forthic-prompt.md documents. Splitting the jobs across two spellings serves both; narrowing the whitelist would serve neither fully.

Structure. is_raw_string_start() plus one branch in transition_from_START, mirroring the existing <<'''…''' marked-redirect lookahead, and a raw flag on transition_from_GATHER_STRING. The triple-quote gather does no escape processing, so the triple-width forms reuse it unchanged and are aliases today. The prompt presents r as an escape hatch for content whose escapes something else has to read — not as a blanket "contains a backslash" rule — and warns that the bare triple form won't keep a backslash.

Compatibility. One narrow break, not a pure addition: words are gathered without breaking on quote characters, so a token spelled r + quote (r'don't') used to lex as a single word and now opens a raw string. No standard-library word is spelled that way, and definition names can't contain quotes, so no definable word is affected. Hence the patch release.

Design. r is the strongest prior available (Python, Rust, Nim, D, Scala) and Forthic already borrowed Python's '''. Backticks were ruled out despite Go and JS — generated Forthic is routinely wrapped in markdown fences. Lowercase only; no composition with the redirect marker. Unlike Python, a raw Forthic string may end in a backslash (r'C:\'C:\), though the same rule means it can't contain its own delimiter.

Testing. 1011 passing, plus build and smoke (including the self-verifying prompt-examples check). Covers the JSON/RUN/path idioms end to end, greedy quote-runs with the prefix, mid-word collision safety, declined redirect composition, empty and unterminated raw strings, and an r-string streamed across every chunk boundary. Disabling the triple-width dispatch fails 9 tests.

Deferred to the breaking release, where the escape flip makes them testable: a streaming chunk boundary landing on a backslash collapses the cumulative value instead of growing it, and StringRedirectRouter.feed() diffs by length alone — so the sink and the stack would silently disagree.

🤖 Generated with Claude Code

https://claude.ai/code/session_01N1Xfa8zk25MrXCZcUEPWY7

Triple-quoted literals were fully raw while single-delimiter strings processed a
seven-character escape whitelist. That split made the escaping regime depend on
how many quote characters were counted: `\'` is correct in `'…'` and ships
verbatim in `'''…'''`, with nothing else distinguishing them.

LLM-authored Forthic does not survive that. Over a 20,000-program production
week, 93% of every backslash occurrence inside a triple-quoted literal is a
`\'` or `\"` the model wrote expecting it to be an escape, and the corruption
reaches users. Replaying the same week through this change drops apostrophe
corruption inside `'''` from 22.97% to 14.91%, and double-quote corruption
inside `"""` from 49.02% to 1.96%.

No language treats `'''` as raw. Python and Groovy are the only languages that
have it and both process escapes; the raw-triple-quote convention belongs to
`"""` in C-family languages where `'` is a character literal and `'''` does not
exist. Forthic paired Python's spelling with Kotlin's semantics.

The whitelist is what keeps this safe rather than a wholesale move to escaped
strings: only \n \t \r \0 \\ \" \' are interpreted, so `\d`, `\w` and `\U`
still stay literal and regexes and Windows paths remain writable. Across the
committed corpus (2,887 literals) exactly two sites change meaning, both in a
gdrive example that ships a literal `\n` into a Google Doc where a newline was
plainly intended.

Escapes resolve before delimiter detection, so an escaped quote is content and
can never close the literal. Unescaped tripling (`'''today'''s plan'''`) still
closes early — this change moves the silent classes, not the loud one.

Three tests asserting the previous raw behaviour are updated; the regex and
path cases they sat beside are untouched and still pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011UUMTq8x3R5KN1pxKJa4ua
@liamfd
liamfd force-pushed the liam/big-7145-triple-quote-escapes branch 2 times, most recently from b1c7a69 to b5210c6 Compare August 14, 2026 15:36
@liamfd liamfd changed the title Add raw string literals (r'…') and keep triple quotes raw Add raw string literals (r'…') ahead of making triple quotes non-raw Aug 14, 2026
Forthic strings carry two jobs that one delimiter cannot serve. Regular strings
interpret a seven-escape whitelist, so an author writing 'today\'s' gets an
apostrophe. Triple-quoted strings carry data, where a backslash has to survive
untouched: JSON payloads and embedded Forthic source parse their own escapes,
and any escape the tokenizer consumes is one the inner layer never sees.

Serving the first job at triple width breaks the second. '''{"a": "x\ny"}'''
JSON> fails on a raw control character, '''{"p": "C:\\Users"}''' JSON> on an
invalid \U, and """ 'a\\nb' """ RUN processes the escape twice. Leaving triple
quotes raw to protect the second leaves the first broken — an LLM writes
'''today\'s plan''' because that is correct one delimiter width narrower, and
the backslash reaches users. Both are paths docs/forthic-prompt.md documents.

An explicit raw form separates the jobs. `r` glued to any opening delimiter
turns escape processing off for that literal: r'…', r"…", r'''…''', r"""…""".

This is a transitional release, not the destination. Triple-quoted strings are
still raw, so the triple-width r forms are aliases today. They exist so data
literals can move before the next release makes '''…''' and """…""" interpret
the whitelist, at which point a bare triple-quoted string stops being a safe
place for a backslash. This is the only window in which both spellings work.

The tokenizer gains is_raw_string_start() and one branch in
transition_from_START, mirroring the existing <<'''…''' marked-redirect
lookahead, plus a raw flag on transition_from_GATHER_STRING that skips its
escape branch. The triple-quote gather does no escape processing, so the
triple-width forms reuse it unchanged. The prompt routes backslash-carrying
strings to the r forms and warns that the bare form will not keep a backslash.

`r` is the strongest prior available (Python, Rust, Nim, D, Scala) and Forthic
already borrowed Python's ''' spelling. Backticks were ruled out despite Go and
JS: generated Forthic is routinely wrapped in markdown fences, where
triple-backtick collides. The prefix is lowercase only and does not compose with
the redirect marker. Unlike Python, a raw Forthic string may end in a backslash
— r'C:\' is C:\ — though the same rule means it cannot contain its own
delimiter.

One narrow break rather than a pure addition: words are gathered without
breaking on quote characters, so a token spelled r + quote (r'don't') used to
lex as a single word and now opens a raw string. No standard-library word is
spelled that way, and definition names cannot contain quotes, so no definable
word is affected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N1Xfa8zk25MrXCZcUEPWY7
@liamfd
liamfd force-pushed the liam/big-7145-triple-quote-escapes branch from b5210c6 to a9fb3f8 Compare August 14, 2026 15:42
@liamfd
liamfd marked this pull request as ready for review August 14, 2026 15:49
@rjose
rjose merged commit 3a7fa1c into forthix:main Aug 14, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants