unescaped quotation marks in macro-processed doc comments

[durka]

It seems like one of the recent fix to macro handling of doc comments (perhaps this) was a bit overzealous in not parsing the string, so that embedded quotation marks don't get escaped. It doesn't show up unless stringify! is called, but then it shows invalid code.

Compare the behavior of this playpen on nightly vs beta/std (ignoring the initial slashes which are due to #23812 and already fixed).

[durka]

Discovered by @birkenfeld

[ghost]

Definitely seems like #26777 broke pretty printing. Important to note that escaping the quotes would do nothing in a raw string anyway, i.e:

#[doc = r" \"It must be Thursday\", said Arthur."]

is still invalid. A short term fix is to bump up the delimiter count in the StrRaw token, but then it can be broken by comments containing "#, e.g:

/// Hello "world"#

would become:

#[doc = r#" Hello "world"#"#]

Maybe # should be escaped when desugaring doc comments?

Not sure if this would break something else though considering # is used everywhere for markdown, example code, issue numbers, etc. Doc comments are complicated!

[barosl]

Oops, I forgot that stringify! can reveal the string representation. My apologies.

I think a workaround to deal with this would be to increase the number of #s in the generated raw string literal. We could look up the number of successive #s in the original string and set the former number to the latter number + 1. In this case, the output would be like:

#[doc = r#" Returns the answer"#] // Successive `#`s == 0, use r#"literal"#
#[doc = r#" "It must be Thursday", said Arthur."#] // Successive `#`s == 0, also use r#"literal"#
#[doc = r###" Ruin everything with "## already "###] // Successive `#`s == 2, use r###"literal"###

Any better ideas?

[barosl]

Also, not directly related to this issue, but is the leading space in the doc string intentional? It seems that the space in the original doc comment (/// comment) was simply preserved in the final doc string (#[doc = r" comment"]), so it didn't lose any information, but it's just not very... pretty.

[ghost]

Sounds like a good solution to me.

As for the leading spaces, I don't think a single space has any significance outside of pretty printing, but obviously leading whitespace in general is significant for markdown formatting.