Readable multi-line quoted string literals

[dbuenzli]

If you want to include precise content with multi-line quoted string literals it forces you to botch its layout: you need to flush-left the content in your sources and this can't align with the start of the literal starting mark.

To give an example assume you have the following content and wish to preserve the exact white space in a quoted string literal:

1. This is the first point
 2. This is the second point

You need to write this as

let my_points = {|1. This is the first point
 2. This is the second point|}

or

let my_points = 
{|1. This is the first point
 2. This is the second point|}

which is not only ugly but also does not convey visually at all the actual textual content of the string. From a source code reading perspective, the design of quoted strings literals is subpar (to not say broken).

With @panglesd we recently went over the definition of code blocks in odoc. This is the outcome (full long design discussion). I think that this is the way multi-line quoted literals should have been defined.

Of course it's a bit late by now. I wonder though if perhaps we could still use a quoted-string-id prefix that triggers a parse according to these definitions. e.g. _ or __,

This would allow for example to write the above content, with natural indentation, as:

let my_points = {_|
                1. This is the first point
                 2. This is the second point
                |_}

or

let my_points = 
  {_|
  1. This is the first point
   2. This is the second point
  |_}

while retaining the exact whitespace content and visual representation.

[gasche]

Maybe {<ident>\ (with nothing else on the line) could be used to mark multiline-whitespace-aware quoted strings. I find this less surprising than changing the lexical conventions based on the quoted string identifier.

(I would use a slightly different specification:

1. It is not obvious to me that the right way to delimit the first column of the box is the placement of the { character, it could also be the first non-whitespace character in the output. (But your choice is okay.)
2. I think you should have been stricter and reject whitespace prefixes that are not equal to the reference prefix (in your case the whitespace before the opening {), in particular different mixes of tabs and spaces. This is a good specification because it is robust to any choice of visual size for tabs. I'm much less convinced by your approach of counting a tab as a single space.
   )

[dbuenzli]

Maybe {<ident>\ (with nothing else on the line) could be used to mark multiline-whitespace-aware quoted strings. I find this less surprising than changing the lexical conventions based on the quoted string identifier.

If that is available in the syntactic space that's fine for me. I find it appealing as it recalls the syntax for breaking string literals across lines. The terminator would remain |} ? I think \} is slightly weird (escaping notation).

1. It is not obvious to me that the right way to delimit the first column of the box is the placement of the { character, it could also be the first non-whitespace character in the output.

No. This doesn't allow you to write arbitrary textual content: you cannot write content with initial whitespace (e.g. swap the two lines of the example in my previous message).

In odoc, allowing initial whitespace is important for all sorts of applications. In quoted strings literals I would say it's even more important to be able to write any kind of textual content.

Given that escapes are not available in quoted string literals, the only way to get around this is to have a syntactic construct external to the string content for defining the first column. Hence the use of opening marker (and the first char of it in order to maximise the number of available columns to write content before hitting 80).

I think you should have been stricter and reject whitespace prefixes that are not equal to the reference prefix (in your case the whitespace before the opening {), in particular different mixes of tabs and spaces. This is a good specification because it is robust to any choice of visual size for tabs. I'm much less convinced by your approach of counting a tab as a single space.

I'm not sure what you mean by reject whitespace prefixes (and what the reference prefix is). If what you meant is to align non-white content sticking on the left of the column of { to the column of {, I agree and I'm fine in doing so since we have a blank page. Incidentally that's how commonmark fenced code blocks work.

In fact that's what I wanted in odoc too. But based on a review of data that would have botched existing renderings too much (see this comment). Compromise was made to preserve existing rendering with that behaviour and emit a warning when these kind of code blocks are detected.

[gasche]

My proposal was that if the {| line uses, say, space-tab-space as leading whitespace, then all other lines considered should start with space-tab-space and this prefix is removed from the data, or it has something else and there is an error/warning.

[dbuenzli]

My proposal was that if the {| line uses, say, space-tab-space as leading whitespace, then all other lines considered should start with space-tab-space

Ah understood. But the catch is that the {| line may not start with whitespace:

let mypoints = {\
               nopoint|}

Except if OCaml lexical conventions want to start to talk about a tabbing model (like commonmark does) I don't think there's a good solution to the problem – well saying that tab is 1 one column is introducing one indirectly.

In any case I don't think that using a column of 1 for tabs is that horrible. If we assume a consistent indentation:

1. If {| and the other lines are all indented by the same tabbed but consistent whitespace all is well visually and content-wise (all column counts match).
2. If {| is indented by non-whitespace (let mypoints = {\), and you indent with a mixture of tab (however your editor defines them but assuming a number of column > 0) and spaces to reach the column of {, by counting tabs as 1, this may only result in a column number that is less or equal than the column of {. In this case your line content is going to be aligned to the column of {. The content will match the visual representation only if it starts with non-whitespace; otherwise you will need to add whitespace until the count reaches the column of { (a punishing discrepancy for having used tabs :–).
   Also it should be stressed that the visual representation may be confusing – but that's something you allow if you buy into tabs in the first place – but the content is in no way dependent on your editor settings.

---

P.S. I'd just like to add that commonmark doesn't have the space vs tab issue in this case because code fences cannot be indented more than 3 spaces of indentation w.r.t. to the current block and a tab moves to the next tab stop which at 4 space of indentation. Hence a code fence marker cannot be introduced by a tab or a mixture of spaces and tabs. If anything you are going to have to indent non-whitespace content on the left of the column of the opening code fence marker ``` by one to up to three spaces.

[copy]

The syntactic space problem could be avoided by using a regular function that does the work at runtime (if really necessary, it could be implemented as an external that gets optimised away):

(* string.mli *)

val trim_indent : string -> string
(** [trim_indent s] detects a common minimal indent of all the input lines,
    removes it from every line and also removes the first and the last lines if
    they are blank.

    Note that blank lines do not affect the detected indent level.

    In case if there are non-blank lines with no leading whitespace characters
    (no indent at all) then the common indent is 0, and therefore this function
    doesn't change the indentation.
*)

(via https://kotlinlang.org/api/core/kotlin-stdlib/kotlin.text/trim-indent.html)

[gasche]

I find this -- having a function -- a rather neat solution, because it is simpler and easier to read than a new lexical construction. I would be sympathetic to a proposal to implement this in the standard library.

[dbuenzli]

How many times will I have to repeat that you may want to have initial whitespace ?

[gasche]

If you want let str = " \nfoo\nbar\n" (start with three spaces), you can write (if I'm reading the specification correctly)

let str = String.trim_indent {|
     
  foo
  bar
|}

(the first line has five spaces: two for indentation and three for the initial whitespace).

[kit-ty-kate]

If you want let str = " \nfoo\nbar\n" (start with three spaces), you can write (if I'm reading the specification correctly)

let str = String.trim_indent {|
     
  foo
  bar
|}

(the first line has five spaces: two for indentation and three for the initial whitespace).

I find this solution rather unfriendly for humans (hiding information in seemingly blank lines sounds like a bad idea) and it also really doesn't work well with editors (and pre-commit hooks) that warn or delete superfluous/trailing whitespaces

[gasche]

Quoted string literals are designed to have the source look as close as possible to its interpretation; this design goal is in tension with making whitespace visually easy to detect. If you want to make it clear that there should be a " \n" line at the beginning, you can also just write " \n" ^ String.trim_indent .... The point of my previous comment is not really that I recommend doing as above, but that I don't think the approach proposed precludes initial whitespace.

[dbuenzli]

So when I want to reindent my code I need to tweak invisible characters in the content of quoted the string. I'm not sure if we are still talking seriously here.

but that I don't think the approach proposed precludes initial whitespace.

There is a misunderstanding here. It's initial whitespace on lines. What you propose is not only user hostile but is also doesn't work for this content (with ␠ standing for U+0020).

␠a
␠b

Which I would like to be able to write naturally, at any indentation level in OCaml source code with an obvious anchor that allows me to easily spot that there is initial whitepace.

Again the gist is:

Given that escapes are not available in quoted string literals, the only way to get around this is to have a syntactic construct external to the string content for defining the first column.

[dbuenzli]

Quoted string literals are designed to have the source look as close as possible to its interpretation;

Btw. by this account we can say that they fail spectacularly at their purpose (see the examples of my first message).

this design goal is in tension with making whitespace visually easy to detect

And that what is being proposed with {\ solves both the original design goal aswell as this aspect.

[Octachron]

Without extending the grammar, we could use a built-in extension point like {%_| or {%code|.

[dbuenzli]

What's difference does it make exactly ?

I have to say I'm quite fond of @gasche's syntactic proposal. I think it easily fits in the brain given existing syntax and it's not noisy.

Besides I expect most people who use quoted literals seriously in the language to switch to them as it fixes them.