verbatim_doc_comment should provide ability to escape newlines with \ like in string literals

[I60R]

Please complete the following tasks

• I have searched the discussions
• I have searched the existing issues

Rust Version

Clap Version

3.0.0.rc7

Minimal reproducible code

/// This is a very long line which I'd like to split in code \
/// but keep wrapped in terminal
/// This text should be on a new line
#[clap(verbatim_doc_comment)]
...

Steps to reproduce the bug with the above code

Actual Behaviour

This is a very long line which I'd like to split in code \
but keep wrapped in terminal
This text should be on a new line

Expected Behaviour

This is a very long line which I'd like to split in code but keep wrapped in terminal
This text should be on a new line

Additional Context

Previously {n} allowed to achieve the expected effect

Debug Output

No response

[epage]

Di you see this as different than #2389?

[I60R]

Yes. That issue was about preserving newlines and verbatim_doc_comment seems to be a proper solution.

This is about escaping newlines away with verbatim_doc_comment using slash

[I60R]

Alternatively, could be the support for {n} restored? It worked perfectly with help_wrap and with it migration from structopt may be easier for some people.

[epage]

I missed that, thanks!

I went back to try to see why {n} was removed but the PR doesn't have an explanation. I've pinged the people involved at that time to see if we can learn from what was done before.

[pksunkara]

Can you try replacing \ with \n and see if it works?

[I60R]

Nope, either with verbatim_doc_comment and without it's rendered as \n without newline being added

[I60R]

@pksunkara \n works when inserted into help attribute e.g. help="string\n" but is stripped when placed in doc comment, so the problem is on derive side I guess?

[pksunkara]

Yup, check the comment I made on that PR. Approach is correct but we need to check for the verbatim_doc_comment attribute first.

[epage]

So to double check what you are ultimately trying to accomplish, you are wanting one line to be able to overflow onto following lines and you want another line immediately after it without a paragraph break, right?

You originally did this with

/// This is a very long line which I'd like to split in code
/// but keep wrapped in terminal {n}
/// This text should be on a new line
...

Your thought is to either implement this with

/// This is a very long line which I'd like to split in code \
/// but keep wrapped in terminal
/// This text should be on a new line
#[clap(verbatim_doc_comment)]
...

or

/// This is a very long line which I'd like to split in code
/// but keep wrapped in terminal \n
/// This text should be on a new line
...

[epage]

btw something I've been meaning to do is to explore the feature set and API design of other CLI parsers to see what we can learn or to help open us up on the problem space.

Click uses escape sequences to customize the help

• \b to turn off wrapping for a paragraph
• \f to cut off the help, leaving the rest for the doc comment

A key difference is that these are just control codes in a doc string that are probably being ignored by the documentation tools while, as pointed out, Rust doc comments are raw strings so escape sequences are ignored.

[I60R]

@epage correct. It seems that \n will be a proper approach because it's very similar to <br> in markdown.

When talking about other CLI parsers, Python's argparse allows custom help formatting via HelpFormatter class, so maybe clap should externalize help preprocessing as well? There could be 4 functions for long_/short_/help/about and they could take Vec<String> from syn plus Vec<(String, String)> of metadata e.g. env., default etc. and users then could implement whatever formatting they want

[pksunkara]

/// This is a very long line which I'd like to split in code
/// but keep wrapped in terminal \n
/// This text should be on a new line

How is this shown on rust docs?

As specified in #2389, our default help text should behave similarly to rust docs. And verbatim_doc_comment allows people to escape that behaviour.

[I60R]

This will be rendered as \n in rustdoc...

Okay, another proposal: in markdown we can place slash at the end of line to preserve newline which is opposite to what slash does in Rust str literals (weird!). Although this feature isn't recommended for use and is relatively unknown, it still a part of markdown specification and thus supported by rustdoc, so

/// This is a very long line which I'd like to split in code
/// but keep wrapped in terminal \
/// This text should be on a new line

Will be rendered by rustdoc as

This is a very long line which I'd like to split in code but keep wrapped in terminal
This text should be on a new line

If we aim to make --help as most close as possible to rustdoc then turning slash followed by newline into {n} replacement should be the most compatible option

[I60R]

So, I've implemented that behavior in #3228, let's continue discussion there

[epage]

In considering the motivating case for this issue being a regression in functionality in clap3 and the relative size of the various proposed replacements for the clap2 behavior, I'm thinking of restoring {n} for the life of clap3, assuming we get markdown support in during the clap3 release. @I60R thoughts?

[epage]

@I60R can you report back if {n} in 3.0.0-rc.11 unblocks you?

[I60R]

Sorry for being silent.

Yes, I like to have {n} during clap3 or until markdown would be ready

[tmccombs]

I want this feature as originally described, but {n} doesn't work, because I need to use verbatim_doc_comment in order to get proper indentation (because without verbatim_doc_comment whitespace gets compressed).

AFAICT currently I either have to use really long lines in the comments with verbatim_doc_comment, or pass it in directly to long_help (i.e. don't use the doc comment functionality). Or I suppose break the line myself, and then if the screen is too narrow, it will wrap weirdly.

Alternatively, having a way to toggle between verbatim and non-verbatim mode. I wonder if it would make sense to switch into verbatim mode inside of a code block delimited with triple backticks (and remove those backticks). Or maybe if a line starts with three or more spaces (2 after removing the initial space), then treat it as a new line, and don't remove the initial whitespace?

Full markdown support isn't really necessary for me here.

[epage]

without verbatim_doc_comment whitespace gets compressed

What do you mean by this? We should only be stripping off a single leading space that would be between the /// and the text.

Full markdown support isn't really necessary for me here.

Except a lot of what you are describing either adds a lot of complexity or is us basically implementing a poor mans version of markdown. A lot of these doc comments people post are because of that gulf; they expect to be able to write markdown and we don't support it.

[tmccombs]

what do you mean by this?

I am referring to:

Strip leading and trailing whitespace from every line, if present.

So if verbatim_doc_comment is true, you lose any indentation at the beginning of the line. Initially, I didn't think there was any way to get that back, but actually I did figure out a way to do it. If I put "{n}" at the beginning of the line, then put the space between that and the text of the line, the whitespace is preserved.

they expect to be able to write markdown and we don't support it.

No, not really. My example isn't even really markdown. It looks something like:

Do something special that requires quite a bit of explanation
that is in a paragraph that should be on multiple
lines in the doc comment, but ideally would be treated
as a single line for wrapping.

Examples:
    --do-something value1
    --do-something 1124

If I were to write that in markdown, I suppose I could put the examples inside triple backquotes. Would that be included in the scope of the markdown support feature?

But as is, it isn't really markdown, just text with some indentation. And it really doesn't need things like headers, emphasis, links, etc. (although such things certainly could be useful in general).

[epage]

Ah, I had missed the trim on non-verbatim lines. That seems odd that we trim leading whitespace.

In general

• We need some format for normalizing the text (paragraphs)
• We need some format for knowing to know how to leave things alone
• There is the general expectation that doc comments are comments and doing a format that is anything else will likely lead to more problems

This leads to us supporting markdown, even if its not the most thorough parser. This is why I created md-rosetta-rs was to look for the cheapest parser that gets things done.

And it really doesn't need things like headers, emphasis, links, etc. (although such things certainly could be useful in general).

I strongly expect headers will be needed. While we are talking about doc comments, I expect people will want to write markdown for others parts as well and we should consider how we can support that for people.