Removal of nre and improvement of re

[dom96]

There isn't time for this in the upcoming release but we need to decide what to do about nre. The consensus between me and Araq is that it should be deprecated and re's API improved to fix issues with it

[andreaferretti]

Can you summarize the main differences between the two?

[Araq]

It provides only a limited number of captures, while the underling library (PCRE) allows an unlimited number.

Not true anymore.

Instead of having one proc that returns both the bounds and substring, it has one for the bounds and another for the substring.

Not an issue since you rarely need both and with the bounds it's easy enough to do input.substr(a, b).

If the splitting regex is empty (""), then it returns the input string instead of following Perl, Javascript, and Java's precedent of returning a list of each character ("123".split(re"") == @["1", "2", "3"]).

Actually it iterates over single character too but with an initial empty string match. Not sure why. Not sure where we claim to be compatible with Java's edge cases.

[juancarlospaco]

Kill NRE 👍

[dom96]

CC @flaviut

[flaviut]

Well it's been a while since I last looked at this, but there are some other things that nre does a lot nicer than re:

• Named capture groups. Reference your groups by name instead of by index. Makes code more readable, especially with complex regexes.
• Error handling. Internal errors are handled in a different way from syntax errors and unicode errors, and error locations are given whenever possible.
• You never have to initialize your own seqs.

I personally also think that the interface is easier to understand and use, but that's pretty subjective.

[timotheecour]

to add to @flaviut :

• nre allows passing a proc to replace, unlike re
• nre makes use of Option which is cleaner

[flaviut]

I keep being referred to this issue, so I'm going to try to organize my thoughts as to why I made nre, and why I think it's important to keep. I'm going to try to go back in time and see if I can describe what my thought process was when I started nre:

There are two fundamental issues I see with re:

• A combinatorial number of methods that exist
• the requirement that I manage the matches array myself

---

There are 3 fundamental operations that can be performed with a string matching library:

• find(): seq[Match]
• match(): Match
• replace(): str
• split(): seq[str]

I'm including match() here because while it is very similar to find("^<pat>$"), it handles multiline strings in a different way.

I also introduce the concept of a Match, which represents a specific application of a regex to a string.

Now, what are the fundamental properties of a Match?

• start and end indices of the match, bounds: (a,b)
• start and end indices of each individual capture, capture_bounds: seq[(a,b)]

These are the fundamental parts of a match, but it's also important to provide the string value of each of those, since it improves code clarity: it's easier to read capture_text than it is to read text[c.a:c.b]. It makes the programmer's intent clearer. So here are two aliases for clarity:

• full match text, match: str
• individual capture text, captures: seq[str]

Let's build a matrix with these operations:

	bounds	match	captureBounds	captures
find	s.find(r).get.matchBounds s.findBounds(r, …)	s.find(r).get.match s.find(r, m: seq[str]); m[0]	s.find(r).get.captureBounds[0] s.find(r, m: seq[slice]); m[1]	s.find(r).get.captures[0] s.find(r, m: seq[string]); m[1]
match	s.match(r).get.matchBounds ???	s.match(r).get.match s.match(r, m: seq[str]); m[0]	s.match(r).get.captureBounds[0] ???	s.match(r).get.captures[0] s.match(r, m: seq[str]); m[1]

With re, what if you want to find something, and you want the bounds of match, but the text of the captures? You need to manually perform your substring. And what if you want to use named matches? You've almost doubled the size of your interface.

nre on the other hand, allows you to perform your operation, and then figure out how you want to use the results.

And that brings us to the second point: nim allows us to elegantly express returning multiple values, but re doesn't take advantage of this feature, so it feels like what a C API feels like:

• you need a large variety of functions that return slightly different things in order to express what you want
• you need to manually allocate your matches array

---

Now that you've read my giant wall of text, I'd like to encourage you to sit down and design your own interface for regular expression stuff. I'd love to hear your thoughts and suggestions! nre may not be the right library, I can absolutely accept that! But I can't accept that re is the best that Nim can do.

[timotheecour]

nice summary. I much prefer nre's API over re's API (for reasons mentioned in this thread) and IMO it'll be easier to build on top of nre rather than other way around. What features are in re that aren't in nre ? /cc @flaviut

misc points:

• find vs match

> find(): seq[Match]
> match(): Match

was that a typo? that's not what's in nre, both return Option[RegexMatch]

I'm including match() here because while it is very similar to find("^$"), it handles multiline strings in a different way.

doesn't seem accurate: nre.match is anchored to the start of the string, so it's find("^<pat>")

• why just "very similar to" and not "same as" ?

• in fact, it's so similar that I feel it's kind of redundant; just find seems enough (being the most flexible) and less confusing (especially since it's anchored at begining, not begin+end)

it handles multiline strings in a different way

not seeing this in docs for nre.match

• IMO all API's should be lazy (eg iterator or operating on 1st match only) instead of greedy (returning a seq) as it's the most flexible/efficient way

[flaviut]

> find(): seq[Match]
> match(): Match

was that a typo? that's not what's in nre, both return Option[RegexMatch]

No, but yes.

I didn't think of "No Match" when writing this, so I did make a mistake here.

But my goal isn't to copy nre here--I'm trying to think of this problem in terms of something like abstract data type. Not any concrete implemenation, just the general concept of a set of operations useful for working with strings and regexes.

doesn't seem accurate: nre.match is anchored to the start of the string, so it's find("^<pat>")

* why just "very similar to" and not "same as" ?

* in fact, it's so similar that I feel it's kind of redundant; just `find` seems enough (being the most flexible) and less confusing (especially since it's anchored at begining, not begin+end)

Multiline patterns mean that ^ can mean either "start of string" or "start of line":

import nre

echo "foo\nbar".match(re"(?m)bar")
echo "foo\nbar".find(re"(?m)^bar$")

None[RegexMatch]
Some(bar)

In single-line mode, they are equvilent. In multi-line mode they are not.

not seeing this in docs for nre.match

The phrasing in the docs was chosen very carefully: it could have said "as if the pattern started with ^", but it says "anchored to the start of the string".

[Araq]

I couldn't use nre for nimgrep and this ship has sailed. nre should become a Nimble package. I agree that re is not nearly as good as it could be, but at least I understand the codebase and continue to improve it. You can argue that re should also move to a Nimble package but then Nim needs testament and testament uses regular expressions so before we do that, we need to remove these dependencies as well.

[flaviut]

I couldn't use nre for nimgrep [...]

Why not? I haven't tested this code, but it's more-or-less what nimgrep would look like with nre:

    let t = find(buffer, pattern, i)
    if t.isNone: break
    inc(line, countLines(buffer, i, t.first-1))

    var wholeMatch = t.get.match

    beforeHighlight()
    inc counter
    if optReplace notin options:
      highlight(buffer, wholeMatch, "", t.get, filename, line, showRepl=false)
    else:
      let r = replace(wholeMatch, pattern, replacement)

[...] and this ship has sailed. nre should become a Nimble package.

I don't disagree, or agree with this. IMO the only thing that's important is that the regex library that ships with the stdlib is the best that Nim can offer. Whether that's some modified version of nre or some version of re is less important. I'd have no problem maintaining nre as a nimble package.

[...], but at least I understand the codebase and continue to improve it.

I'm disapointed to hear this, since I try to make make my code as easy to read and write as possible.

If you can find time, can you provide some more specific feedback here? What changes could be made to make it easier to maintain?

You can argue that re should also move to a Nimble package but then Nim needs testament and testament uses regular expressions so before we do that, we need to remove these dependencies as well.

I won't make this argument because regexes are so useful that I think it's important that the stdlib ships something to handle them.

[Araq]

If you can find time, can you provide some more specific feedback here? What changes could be made to make it easier to maintain?

Well nothing, I wrote re and not nre and that's what makes the difference for me. re also has fewer lines of code, I think.

IMO the only thing that's important is that the regex library that ships with the stdlib is the best that Nim can offer. Whether that's some modified version of nre or some version of re is less important. I'd have no problem maintaining nre as a nimble package.

Problem is, this is not how we can develop the stdlib realistically, the "best that Nim can offer" is frequently written by somebody else, maintained as a Nimble package, see nitely's native Nim regex implementation that doesn't depend on PCRE.

IMO what we need for the stdlib is something that is small, re.nim accomplished that, then bugs were found and instead of fixing the bugs nre was written with a whole different API.

[timotheecour]

"moving to a Nimble package" doesn't work for code in stdlib(or nim compiler) that needs features in nre. As was previously pointed out in this issue as well as in nim-lang/Nim#8150, nre has a better API than re and it will be easier to port over the missing features in nre (found in re) rather than the other way around.
As for external nimble package, we can focus efforts on improving nim-regex (and add to nim-regex some features from nre)