strconv: add QuotedPrefix

[dsnet]

The existing Unquote function unescapes a Go string assuming that the entirety of input string is the escaped the string. However, in many parsing applications, we have a quoted string followed by some amount of unconsumed input, the presence of arbitrary characters after the quoted string currently breaks the Unquote function.

I propose adding:

// UnquoteLeading interprets the start of s as a single-quoted, double-quoted, or backquoted Go string literal.
// Any subsequent characters after the terminating quote are returned as rem.
func UnquoteLeading(s string) (out, rem string, error)

This functionality may simplify a number of standard packages that implement their own logic to determine the end of a quoted string so that they can pass the correctly sized string to strconv.Unquote:

• fmt/scan.go
• text/template/parse/lex.go
• reflect/type.go

[mvdan]

I think this proposal makes sense; having to find the end of the quote manually means reimplementing a basic version of Unquote.

I also wonder if the input should be a []byte; I realise the package is called strconv, but the use case is precisely about reading and buffering, where one almost always has a byte slice. The fmt and reflect examples you show have to explicitly convert to string before calling strconv, for example. I think the result is less important, as it's possible to construct the result with a strings.Builder.

[rsc]

It may be that the operation to expose is "find me the end of this string" and not "also Unquote it".
I was discussing recently with Austin about adding a similar operation to regexp/parse, but I was thinking it would be separate from actually parsing. You don't always want to parse it, but you always need to identify a boundary.

[mvdan]

That sounds more flexible, with the downside that one would end up looping over the input bytes twice. That already happens today, so perhaps the cost isn't significant enough to matter.

[dsnet]

Given that many strings do not have any escape characters, the operation could report whether the string had any escape characters. If not, the user could trivially truncate off the leading and trailing quote characters and avoid a second pass.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

So if we do the string-end-finder, I guess the API would be:

package strconv

// QuotedPrefix returns the quoted string (as defined by Quote and Unquote) at the prefix of s.
// If s does not start with a valid quoted string, QuotedPrefix returns an error.
func QuotedPrefix(s string) (string, error)

?

Reporting the presence of absence of escape characters seems like premature optimization.

[dsnet]

Wouldn't you need to know where the prefixed quoted string ended so that the caller can start parsing what comes afterwards?

Reporting the presence of absence of escape characters seems like premature optimization.

I agree. In my all my use-cases, performance was not paramount. Also, this loop does not seem hard to write in the extremely performance sensitive cases.

[ianlancetaylor]

Wouldn't you need to know where the prefixed quoted string ended so that the caller can start parsing what comes afterwards?

You can call the len function on the result.

[dsnet]

Ah yes, I apologize. I missed the part where it "returns the quoted string".

[dsnet]

I feel like an API that simply returns the length of a valid quoted string (and the entire length otherwise) is simpler to use.

Suppose we had:

// QuotedPrefixLen returns the length of the quoted string (as defined by Unquote) at the prefix of s.
// If s does not start with a valid quoted string, QuotedPrefix returns len(s).
func QuotedPrefixLen(s string) int

For example, if I wanted to parse the following text:

"string1" "string2" "string3"

the code would look like:

in = strings.TrimSpace(in)
for len(in) > 0 {
    n := strconv.QuotedPrefixLen(in)
    s, err := strconv.Unquote(in[:n])
    if err != nil {
        ...
    }
    ... // make use of s
    in = strings.TrimSpace(in[n:])
}

Alternatively, with the QuotedPrefix API, you have to do an additional error check:

in = strings.TrimLeft(in, " ")
for len(in) > 0 {
    s1, err := strconv.QuotedPrefix(in)
    if err != nil {
        ...
    }
    s2, err := strconv.Unquote(s1)
    if err != nil {
        ... // technically this will never happen, but it's not obvious to reviewers
    }
    ... // make use of s2
    in = strings.TrimLeft(in[len(s1):], " ")
}

That said, I'm still okay with QuotedPrefix as its better than nothing.

[rogpeppe]

I feel like an API that simply returns the length of a valid quoted string (and the entire length otherwise) is simpler to use.

Suppose we had:

// QuotedPrefixLen returns the length of the quoted string (as defined by Unquote) at the prefix of s.
// If s does not start with a valid quoted string, QuotedPrefix returns len(s).
func QuotedPrefixLen(s string) int

For example, if I wanted to parse the following text:

"string1" "string2" "string3"

the code would look like:

in = strings.TrimSpace(in)
for len(in) > 0 {
    n := strconv.QuotedPrefixLen(in)

Would QuotedPrefixLen return 0 if there's an escaping error inside the quoted prefix?

I guess if that's the case, you could still call Unquote on the full string and it'll produce the expected error, so it'll work out OK.

[dsnet]

I struggled between having it return 0 or len(s) on error situations. The advantage of returning len(s) is that naive slicing of the input would pass an invalid string to strconv.Unquote, which would then produce a good error message. The downside is that you can't distinguish between a valid string that happens to be the full length or an invalid one.