string replace: Regular expression substitute error: requested value is not set

[planet36]

Version info

fish --version

fish, version 3.1.2

echo $version

3.1.2

uname -a

Linux archvm 5.8.10-arch1-1 #1 SMP PREEMPT Thu, 17 Sep 2020 18:01:06 +0000 x86_64 GNU/Linux

echo $TERM

xterm-256color

Description

string replace prints an error when I try to substitute a back reference to an optional capture group that was not matched.

Steps to reproduce

Fish

echo 'b'   | string replace -r -- '(q?)b\1' 'FOO $1 BAR'
echo 'qb'  | string replace -r -- '(q?)b\1' 'FOO $1 BAR'
echo 'qbq' | string replace -r -- '(q?)b\1' 'FOO $1 BAR'
echo

echo 'b'   | string replace -r -- '(q)?b\1' 'FOO $1 BAR'
echo 'qb'  | string replace -r -- '(q)?b\1' 'FOO $1 BAR'
echo 'qbq' | string replace -r -- '(q)?b\1' 'FOO $1 BAR'
echo

echo 'b'   | string replace -r -- '(q)?b\1?' 'FOO $1 BAR'
echo 'qb'  | string replace -r -- '(q)?b\1?' 'FOO $1 BAR'
echo 'qbq' | string replace -r -- '(q)?b\1?' 'FOO $1 BAR'
echo

Output

FOO  BAR
qFOO  BAR
FOO q BAR

b
qb
FOO q BAR

string replace: Regular expression substitute error: requested value is not set
FOO q BAR
FOO q BAR

Perl

echo 'b'   | perl -ne 's/(q?)b\1/FOO $1 BAR/; print'
echo 'qb'  | perl -ne 's/(q?)b\1/FOO $1 BAR/; print'
echo 'qbq' | perl -ne 's/(q?)b\1/FOO $1 BAR/; print'
echo

echo 'b'   | perl -ne 's/(q)?b\1/FOO $1 BAR/; print'
echo 'qb'  | perl -ne 's/(q)?b\1/FOO $1 BAR/; print'
echo 'qbq' | perl -ne 's/(q)?b\1/FOO $1 BAR/; print'
echo

echo 'b'   | perl -ne 's/(q)?b\1?/FOO $1 BAR/; print'
echo 'qb'  | perl -ne 's/(q)?b\1?/FOO $1 BAR/; print'
echo 'qbq' | perl -ne 's/(q)?b\1?/FOO $1 BAR/; print'
echo

Output

FOO  BAR
qFOO  BAR
FOO q BAR

b
qb
FOO q BAR

FOO  BAR
FOO q BAR
FOO q BAR

I expected string replace to behave similar to perl because the docs mention "Perl-compatible" regular expressions.

[faho]

I expected string replace to behave similar to perl because the docs mention "Perl-compatible" regular expressions.

"perl-compatible" as in "uses PCRE2" (which also explicitly calls itself "Perl-compatible").

That's where the error comes from, so I don't think there is anything for us to do here.

[krobelus]

@planet36 you can try reporting this to the PCRE bug tracker

[planet36]

@planet36 you can try reporting this to the PCRE bug tracker

https://bugs.exim.org/show_bug.cgi?id=2645

[planet36]

Here's a simpler example

echo 'az'   | string replace -r -- 'a(b.+)?z' 'a:$1z'
echo 'abz'  | string replace -r -- 'a(b.+)?z' 'a:$1z'
echo 'abcz' | string replace -r -- 'a(b.+)?z' 'a:$1z'

Output

string replace: Regular expression substitute error: requested value is not set
abz
a:bcz

[faho]

Okay, it turns out there's an option for this:

PCRE2_SUBSTITUTE_UNSET_EMPTY causes unset capture groups (including unknown groups when PCRE2_SUBSTITUTE_UNKNOWN_UNSET is set) to be treated as empty strings when inserted as described above. If this option is not set, an attempt to insert an unset group causes the PCRE2_ERROR_UNSET error. This option does not influence the extended substitution syntax described below.