RFC: Syntax for raw string literals

[lilyball]

A raw string literal is a string literal that does not interpret any embedded sequence, meaning no backslash-escapes. A lot of languages (certainly most that I've used) support some syntax for raw string literals. They're useful for embedding any string that wants to have a bunch of backslashes in it (typically because the function the string is passed to wants to interpret them itself), such as regular expressions. Unfortunately, Rust does not have a raw string literal syntax.

There's been a discussion on the mailing list for the past few days about this. I will try to put a quick summary here.

There's two questions at stake. The first is, should Rust have a raw string literal syntax? The second is, if so, what particular syntax should be used? I think the answer to the first is definitely Yes. It's useful enough, and has enough overwhelming precedence in other languages, that we should add it. The question of concrete syntax is the harder one.

The syntaxes that have been proposed so far, along with their Pros and Cons:

1. C++11 syntax, e.g. R"delim(raw text)delim".

   Pros:

   • Reasonably straightforward
   • Can embed any character sequence

   Cons:

   • Syntax is slightly complicated (editorial note: I think any syntax that's flexible enough to contain any character is going to be considered slightly complicated).

2. Python syntax, e.g. r"foo"

   Pros:

   • Simple syntax

   Cons:

   • Can't embed any character sequence.
   • Python's implementation has really wacky handling of backslash escapes in conjunction with the quote character. Even reproducing that behavior does not allow for embedding any sequence, as r"foo\"" evaluates to the string foo\" (with the literal backslash).

3. D syntax, e.g. r"raw text", raw text, or q"(raw text)"/q"delim\nraw text\ndelim"

   Pros:

   • Can embed any character sequence (with the third variant)

   Cons:

   • The first two forms aren't flexible enough, and the third form is a bit confusing. The delimiter behaves differently depending on whether it's a "nesting" delimiter (one of ([<{), another token, or an identifier.

4. C#/SQL/something else, using a simple raw string syntax such as r"text" where doubling up the quote inserts a single quote, as in r"foo""bar"

   Pros:

   • Simple syntax

   Cons:

   • Does not reproduce verbatim every character found in the source sequence, which makes it slightly harder/more confusing to read, and more annoying to do things like pasting a raw string into your source file (e.g. raw HTML).

5. Perl quote-like operators, e.g. q{text}. Unfortunately, most viable delimiters will result in an ambiguous parse.

6. Ruby quote-like operators, e.g. %q{text}. Unfortunately, this also is ambiguous (with the % token).

7. Lua syntax, e.g. [=[text]=]

   Pros:

   • Simple syntax
   • Can embed any character sequence

   Cons:

   • Syntax looks decidedly non-string-like
   • Custom delimiters are limited to sequences of =
   • Alex Chrichton opined that seeing println!([[Hello, {}!]], "world") in an introduction to Rust would be awfully confusing (see previous point about being non-string-like).

8. Go syntax, e.g. raw text. This is one of the variants of D strings as well

   Pros:

   • Simple syntax

   Cons:

   • Cannot embed any character sequence (notably, cannot embed backtick)
   • It's difficult or impossible to embed backticks in a markdown code sequence, which will make it awkward to use raw strings in markdown editors. May also be confusing with the usage of foo in doc comments.

9. A new syntax using ASCII Control characters STX and ETX

   Pros:

   • I don't think there are any

   Cons:

   • Can't type the keys on any keyboard
   • Text editors probably won't render the characters correctly either
   • Can't technically embed any character sequence, because ETX cannot be embedded, but in fairness it can embed any printable sequence.

10. A syntax proposed over IRC is delim"raw text"delim.

    Pros:

    • Can embed any character

    Cons:

    • Unusual syntax with no precedent in other languages. Functionally identical to C++11 syntax.
    • Hard to type in Markdown editors

Some form of Heredoc syntax was also suggested, but heredocs are really primarily concerned with embedding multiline input, not raw input. They also have issues around dealing with indentation and the first/last newline.

During this discussion, only two Rust team members (that I'm aware of) chimed in. Alex Chricton raised issues with the Lua syntax, and threw out the suggestion of Go's syntax, though only as something to consider rather than a recommendation. Felix Klock expressed a preference for C++11 syntax, and more generally stated that he wants a syntax with user-delimited sequences. There was also at least one community member in favor of C++11 syntax.

My own preference at this point is for C++11 syntax as well. At the very least, something similar to C++11 syntax, that shares all of its properties, but there seems to be no value in inventing a new syntax when there's precedent in C++11.

[thestinger]

I think functionality equivalent to the C++11 syntax is best, but ideally not as noisy. We also need to consider how text editor syntax files will handle it, but I don't think it will be too much of a problem.

[lilyball]

@thestinger Do you have any suggestions? Functionally equivalent to C++11 requires something of the form <start token><user-supplied delimiter><delimiter end token><raw text><delimiter start token><user supplied delimiter><end token>. In C++11 the <start token> is R", the <delimiter end token> is (, the <delimiter start token> is ), and <end token> is ". I don't think you can remove any of these components without breaking functionality, and I don't think you can adjust the values to produce something less noisy.

The only adjustment I can think of would be to remove <delimiter start/end token> and instead require that the <user supplied delimiter> includes an appropriate punctuation to end it (and to start the close sequence), but if anything that makes it more confusing, not less.

How about R<space><user-supplied delimiter>"<raw text>"<user-supplied delimiter> and R"<raw text>" for the case where a simple " delimiter is sufficient.

R"c:\some\path\"
R eos"raw text"eos

[lilyball]

@stevenashley The lexer will see the R<space> and tokenize that as an identifier R. Trying to look ahead past spaces is, at best, highly confusing.

Ah, of course. I can't think of a substitute for <space> that would simultaneously look nice and parse well. Consider my proposal retracted.

[Kimundi]

How about this: r"" syntax, with the option to pad the string on both ends with #:

foo  ==  r"foo"
fo"o ==  r#"fo"o"#
##   ==  r###"##"###

As far as I know we don't allow # in an expression context, it's only valid as part of the attribute syntax, so this should work.
Heck, it would even be ok in attributes themself, I think:

#[foo = r##"test"##];

Alternatively, we could also throw away the r token itself and say that any number of # followed by " starts an raw string literal:

let regex = ##"[/s]+"##;

Or we make both forms valid: r"" for short raw strings, ##""## as alternative to cover every possible string.

[lilyball]

@Kimundi That looks like a reinvention of Lua's [==[foo]==] syntax. It's certainly workable, but it shares the same problems that Lua syntax does (as pointed out by @alexcrichton).

[alexcrichton]

It's probably also worth re-mentioning the various use-cases which have a desire for some sort of syntax that is not what we have today:

1. Regular expressions. These contain lots of backslashes and normally escapes aren't even really that necessary. If we used the normal string syntax, everything would have to be double-escaped which is a pain. The main stickler about this desired syntax is that this would want to be very usable (in the sense that it shouldn't be a pain to write/read read the strings of regular expressions, at least no more than it already is).
2. Literal windows paths. Perhaps these should be done in a different manner to be portable, but regardless having to escape the \ character is a real pain. As with regular expressions, in theory the string syntax isn't difficult to read.
3. Giant blobs of raw text, such as formatting an HTML document (like what rustdoc does right now). This is different from regular expressions in that I they don't need to be so easily readable (because the body of the text is normally very large), so the custom delimiters surrounding the text I believe would be find in this case.
4. format! string directives. Right now it's a pain to print a \ character because you have to type println!("\\\\"). As with regular expressions though, this should be easy to read and easy to use (because it may be fairly common). Perhaps this should use a different escape though, which would make this irrelevant.

Those are the use cases that I could think of, others may have more

Would case 4 become println!(R"(\\)") ?

[alexcrichton]

I suppose under the C++ syntax that's what it would be, which is arguably just as confusing as four backslashes.

[Kimundi]

@kballard I would say it's better than Luas syntax here.

• It has the same advantage of being able to delimit any text.
• Only being limited to # is not a problem, you can still find a delimiter sequence for any input.
• The default case r"" has very low typing overhead, and looks very similar to a regular string literal, no confusion about meaning.

Looking at @alexcrichton's use cases:

1. Regular expressions: r"([^:]*):(\d+):(\d+): (\d+):(\d+) (.*)$".match_groups()
2. Windows paths: r"C:\Program Files\rust\bin\rust.exe".to_path()
3. format! strings: println!(r"\\");
4. Blobs of text:

static MARKDOWN: &'static str = r###"
## Scope

This is an introductory tutorial for the Rust programming language. It
covers the fundamentals of the language, including the syntax, the
type system and memory model, generics, and modules. [Additional
tutorials](#what-next) cover specific language features in greater
depth.

This tutorial assumes that the reader is already familiar with one or
more languages in the C family. Understanding of pointers and general
memory management techniques will help.
"###;

[alexcrichton]

Those all look totally reasonable to me.

Me also. I prefer Kimundi's proposed syntax over C++11 syntax. Nicely done.

[huonw]

(All of these mean the token language is no longer regular, right?)

[lilyball]

@Kimundi: Regular expression:

r##"(\w+)   # match word chars
    "[^"]*" # followed by a quoted string
    (\d+)   # followed by digits"##.flag("x").match_groups();

# just looks like a very odd character here.

That said, I am not as adverse to this syntax as I was initially. While I think it looks weird, and it would feel weird every time I type it, I would be ok with using it.

@huonw I believe you are correct. Is that particularly important?