proposal: regexp/syntax: relax named subexpressions (capture groups) name restrictions

[mitar]

Currently, Go regular expression language supports named subexpressions (also known as named capture groups), i.e., (?P<name>re). The topic of this proposal is what are restrictions on name. I have not found anything documented but from looking at the code it looks like parsing parses everything from < up to the first next > and then validates name using isValidCaptureName which has comment:

// isValidCaptureName reports whether name
// is a valid capture name: [A-Za-z0-9_]+.
// PCRE limits names to 32 bytes.
// Python rejects names starting with digits.
// We don't enforce either of those.

I would like to suggest that this check is relaxed and that Go allows all characters except > (but I would be also OK with less relaxation). Capture names are already not fully compatible with PCRE nor Python, so I think they could be relaxed further.

Motivation

I made a simple tool to convert text to JSON by providing a regexp. How this conversion happens is provided as the name of the capture group. The basic idea is (?P<foo>.*) would create a filed foo in JSON with the matched value. But I also want some transformations of matched values (parsing ints, floats, dates, supporting arrays). For that I had to use a very awkward syntax with __ (double underscore) to separate arguments and ___ (triple underscore) to separate operators. E.g.: (?P<foo__bar___int>.*) would parse the value into int and store it into {"foo": {"bar": <int>}}. I think some standard syntax where I could use dots like (?P<foo.bar>.*) and arrays like (?P<foo[]>.*) and parenthesis and arguments like (?P<date("2006-01-02T15:04:05Z07:00")>.*) would be much nicer. The last example shows also another issue with current restrictions on names: I cannot really pass arbitrary date parsing layout but I can support only predefined ones. Similarly, I cannot pass location for time parsing as Europe/Ljubljana because / is not allowed.

I know this is maybe looks like a niche use case, but to me the idea really opened a new way of working with data, similarly how struct tags enable various ways on how data is converted into structs, regexp could also allow that so that both what text to extract and how to map that to a struct could be all in the same string (which can then be passed to the program as CLI argument).

[AlexanderYastrebov]

I think some standard syntax where I could use dots like (?P<foo.bar>.) and arrays like (?P<foo[]>.) and parenthesis and arguments like (?P<date("2006-01-02T15:04:05Z07:00")>.*) would be much nicer.

FWIW you may pre-process regexp string, map subexpressions to generated valid names, rewrite regexp and resolve names back after matching during evaluation.
E.g. for (?P<foo.bar>.*) (?P<baz.qux>.*) you rewrite it into (?P<subexp_0>.*) (?P<subexp_1>.*) and keep mapping {"subexp_0": "foo.bar", "subexp_1": "baz.qux"} then after matching regexp against input map subexp_0 and subexp_1 back to foo.bar and baz.qux respectively and evaluate them as you please.

[mitar]

I am not sure if Go exposes regexp parsing at that level? Or are you suggesting I parse it and replace those myself? I think then I would have to reimplement at least some level of regexp parsing?

[AlexanderYastrebov]

I am not sure if Go exposes regexp parsing at that level?

You can parse regexp with regexp ;) https://go.dev/play/p/pYhPKGVby3D

[mitar]

Oh, I do not think it is so simple. Such replacement would fail on regexp like (contrived) foo\?P<bar>baz, which does not use capture group at all, and it escapes ? so it simply matches a static string. But your code would replace bar with something else, making the regexp not match at all.

[junyer]

If you really want to embed a domain-specific language within regular expressions, you can do so with (* … *) comments that your package recognises and removes before calling into the regexp package.

[mitar]

I would prefer not to have to pre-parse regex myself because then I also have to implement my own escaping mechanism and users will have to think about those two layers of syntax.

(My understanding is that there is no existing comment syntax in Go regexp I could reuse here and that I could ideally even access comments after parsing. So your suggestion is the same as one by @AlexanderYastrebov and I would like to avoid such string munging. This is why I made this proposal.)

[junyer]

The thing is that embedding a domain-specific language within regular expressions necessarily involves two layers of syntax; any users that you might have acquired over the past two weeks are already being forced to think about two layers of syntax.

Moreover, the correct approach is not to abuse regular expressions, but to enable users to write parsers in a domain-specific language suited to writing parsers. With that in mind, it's unclear why the regexp package should facilitate such a use case.

[mitar]

Moreover, the correct approach is not to abuse regular expressions, but to enable users to write parsers in a domain-specific language suited to writing parsers.

But they are already familiar with regular expressions and I think this is a huge advantage over asking them to learn another language.

With that in mind, it's unclear why the regexp package should facilitate such a use case.

I understand if you are reluctant to do so. To me it really looks like something which can enable interesting exploration of new ideas. Like how struct tags did.

Probably preprocessing regex is the right approach then. But I would really like to avoid having to maintain my own regexp parsing just to find/replace capture group names. With golang regexp not supporting lookbehinds to me it looks like it is not possible to properly do what @AlexanderYastrebov with just regexp which would handle also regexp escaping (i.e., not replace a capture group which just looks like a capture group but in fact it is not because something is escaped). Am I mistaken here?

[junyer]

I would handle escaping by looping over the runes: if the rune is \, consume and output the rune and the next rune; else if the rune is ( and the next runes are ?P<, consume the runes and the following runes until (and including) >, then output (?P<, the generated capturing group name and >; else, consume and output the rune.

[seankhliao]

re2 allows more unicode now: google/re2@6a99418