/(.(?2))((?<=(?=(?1)).))/ hangs and eats all available RAM

[p5pRT]

Migrated from rt.perl.org#126182 (status was 'resolved')

Searchable as RT126182$

[p5pRT]

From victor@drawall.cc

Created by @Grimy

How to reproduce
----------------

perl5.23.4 -e '/(.(?2))((?<=(?=(?1)).))/'

Expected behavior
-----------------

Perl should immediately die with the following diagnostic​:

Infinite recursion in regex at -e line 1.

This is the current behavior of almost everything that would cause a regex to
never complete, creating the expectation that the regex engine is safe from
this kind of problem.

Actual behavior
---------------

Perl becomes unresponsive, allocating more and more RAM

Perl Info

Flags:
    category=core
    severity=wishlist

Site configuration information for perl 5.23.4:

Configured by grimy at Tue Sep 22 21:18:14 CEST 2015.

Summary of my perl5 (revision 5 version 23 subversion 4) configuration:
  Commit id: 2d9b5f101563ac9fee41e6ca496f79db6222d2e3
  Platform:
    osname=linux, osvers=4.0.7-2-arch, archname=x86_64-linux
    uname='linux localhost 4.0.7-2-arch #1 smp preempt tue jun 30
07:50:21 utc 2015 x86_64 gnulinux '
    config_args='-ds -e -Dusedevel'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-fwrapv -fno-strict-aliasing -pipe
-fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-fwrapv -fno-strict-aliasing -pipe
-fstack-protector-strong -I/usr/local/include'
    ccversion='', gccversion='5.1.0', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8,
byteorder=12345678, doublekind=3
    d_longlong=define, longlongsize=8, d_longdbl=define,
longdblsize=16, longdblkind=3
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib
/usr/lib/gcc/x86_64-unknown-linux-gnu/5.1.0/include-fixed /usr/lib
/lib/../lib /usr/lib/../lib /lib /lib64 /usr/lib64
    libs=-lpthread -lnsl -lnm -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc
-lgdbm_compat
    perllibs=-lpthread -lnsl -lnm -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.21.so, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.21'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib
-fstack-protector-strong'



@INC for perl 5.23.4:
    /usr/local/lib/perl5/site_perl/5.23.4/x86_64-linux
    /usr/local/lib/perl5/site_perl/5.23.4
    /usr/local/lib/perl5/5.23.4/x86_64-linux
    /usr/local/lib/perl5/5.23.4
    /usr/local/lib/perl5/site_perl
    .


Environment for perl 5.23.4:
    HOME=/home/grimy
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/grimy/bin:/home/grimy/.nvim/scripts:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/opt/plan9/bin
    PERL_BADLANG (unset)
    SHELL (unset)

[p5pRT]

From @khwilliamson

I am unable to reproduce this. I've tried on both -DEBUGGING builds, and not, and in 5.22.2

The regex compiles in all builds I tried it as

Final program​:
  1​: OPEN1 (3)
  3​: REG_ANY (4)
  4​: GOSUB2[+5] (7)
  7​: CLOSE1 (9)
  9​: OPEN2 (11)
  11​: IFMATCH[-1] (23)
  13​: IFMATCH[0] (20)
  15​: GOSUB1[-14] (18)
  18​: SUCCEED (0)
  19​: TAIL (20)
  20​: REG_ANY (21)
  21​: SUCCEED (0)
  22​: TAIL (23)
  23​: CLOSE2 (25)
  25​: END (0)
minlen 1
r->extflags​: NO_INPLACE_SUBST
r->intflags​: [none-set]

--
Karl Williamson

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From victor@drawall.cc

My bad, my “How to reproduce section” is wrong. When matched against
an empty string, the regex will fail immediately, as expected. The
problems only start when it is applied to a non-empty string, as in :

perl -e '"a"=~/(.(?2))((?<=(?=(?1)).))/'

I’ve just reproduced it on the latest blead
(f83e001).

I’d recommend doing a `ulimit -Sv 1048576` or some such before running
the command, to prevent it from getting out of control.

[p5pRT]

From @demerphq

On 14 October 2015 at 08​:05, Victor ADAM <victor@​drawall.cc> wrote​:

My bad, my “How to reproduce section” is wrong. When matched against
an empty string, the regex will fail immediately, as expected. The
problems only start when it is applied to a non-empty string, as in :

perl -e '"a"=~/(.(?2))((?<=(?=(?1)).))/'

I’ve just reproduced it on the latest blead
(f83e001).

I’d recommend doing a `ulimit -Sv 1048576` or some such before running
the command, to prevent it from getting out of control.

This should trigger an infinite recursion error.

yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

[p5pRT]

From victor@drawall.cc

Yes, it *should* trigger an infinite recursion error, but it actually doesn’t.

[p5pRT]

From @khwilliamson

On 10/14/2015 03​:11 AM, Victor ADAM wrote​:

Yes, it *should* trigger an infinite recursion error, but it actually doesn’t.

Thanks for the ulimit hint.

This is an area of the engine that I have never looked at, so it would
take me a while to get up to speed to figure out how to fix it. Here's
the first bit of the output of -Dr execution. I see why the current
check doesn't work in this situation, but am unsure of how one might fix
it without breaking other things​:

Matching REx "(.(?2))((?<=(?=(?1)).))" against "a"
  0 <> <a> | 1​:OPEN1(3)
  0 <> <a> | 3​:REG_ANY(4)
  1 <a> <> | 4​:GOSUB2[+5](7)
  1 <a> <> | 9​: OPEN2(11)
  1 <a> <> | 11​: IFMATCH[-1](23)
  0 <> <a> | 13​: IFMATCH[0](20)
  0 <> <a> | 15​: GOSUB1[-14](18)
  0 <> <a> | 1​: OPEN1(3)
  0 <> <a> | 3​: REG_ANY(4)
  1 <a> <> | 4​: GOSUB2[+5](7)
  1 <a> <> | 9​: OPEN2(11)
  1 <a> <> | 11​: IFMATCH[-1](23)
  0 <> <a> | 13​: IFMATCH[0](20)
  0 <> <a> | 15​: GOSUB1[-14](18)
  0 <> <a> | 1​: OPEN1(3)
  0 <> <a> | 3​: REG_ANY(4)
  1 <a> <> | 4​: GOSUB2[+5](7)
  1 <a> <> | 9​: OPEN2(11)
  1 <a> <> | 11​: IFMATCH[-1](23)
  0 <> <a> | 13​: IFMATCH[0](20)
  0 <> <a> | 15​: GOSUB1[-14](18)
  0 <> <a> | 1​: OPEN1(3)
  0 <> <a> | 3​: REG_ANY(4)
  1 <a> <> | 4​: GOSUB2[+5](7)
  1 <a> <> | 9​: OPEN2(11)
  1 <a> <> | 11​:
IFMATCH[-1](23)
  0 <> <a> | 13​:
IFMATCH[0](20)
  0 <> <a> | 15​:
GOSUB1[-14](18)

[p5pRT]

From @demerphq

On 14 October 2015 at 18​:01, Karl Williamson <public@​khwilliamson.com> wrote​:

On 10/14/2015 03​:11 AM, Victor ADAM wrote​:

Yes, it *should* trigger an infinite recursion error, but it actually
doesn’t.

Thanks for the ulimit hint.

This is an area of the engine that I have never looked at, so it would take
me a while to get up to speed to figure out how to fix it.

I have a fix. (Bit vector of visited nodes).

Here's the first
bit of the output of -Dr execution. I see why the current check doesn't
work in this situation, but am unsure of how one might fix it without
breaking other things

Exactly. I am still investigating. Might take me a few days, but ill get there.

Yves

[p5pRT]

From @khwilliamson

On 10/14/2015 11​:09 AM, demerphq wrote​:

On 14 October 2015 at 18​:01, Karl Williamson <public@​khwilliamson.com> wrote​:

On 10/14/2015 03​:11 AM, Victor ADAM wrote​:

Yes, it *should* trigger an infinite recursion error, but it actually
doesn’t.

Thanks for the ulimit hint.

This is an area of the engine that I have never looked at, so it would take
me a while to get up to speed to figure out how to fix it.

I have a fix. (Bit vector of visited nodes).

Here's the first
bit of the output of -Dr execution. I see why the current check doesn't
work in this situation, but am unsure of how one might fix it without
breaking other things

Exactly. I am still investigating. Might take me a few days, but ill get there.

Yves

Yves++

[p5pRT]

From @khwilliamson

On 10/14/2015 11​:09 AM, demerphq wrote​:

On 14 October 2015 at 18​:01, Karl Williamson <public@​khwilliamson.com> wrote​:

On 10/14/2015 03​:11 AM, Victor ADAM wrote​:

Yes, it *should* trigger an infinite recursion error, but it actually
doesn’t.

Thanks for the ulimit hint.

This is an area of the engine that I have never looked at, so it would take
me a while to get up to speed to figure out how to fix it.

I have a fix. (Bit vector of visited nodes).

Here's the first
bit of the output of -Dr execution. I see why the current check doesn't
work in this situation, but am unsure of how one might fix it without
breaking other things

Exactly. I am still investigating. Might take me a few days, but ill get there.

Yves

How is this coming?

[p5pRT]

From @demerphq

I'm really sorry, a series of personal issues, family in hospital type
things, have kept me from finishing this. I hope to find time at some
point, but I cant say when.

If it would help I will try to find time to upload what
work-in-progress I have. Otherwise I will do my best to get to it when
time permits.

Yves

On 26 December 2015 at 06​:24, Karl Williamson <public@​khwilliamson.com> wrote​:

On 10/14/2015 11​:09 AM, demerphq wrote​:

On 14 October 2015 at 18​:01, Karl Williamson <public@​khwilliamson.com>
wrote​:

On 10/14/2015 03​:11 AM, Victor ADAM wrote​:

Yes, it *should* trigger an infinite recursion error, but it actually
doesn’t.

Thanks for the ulimit hint.

This is an area of the engine that I have never looked at, so it would
take
me a while to get up to speed to figure out how to fix it.

I have a fix. (Bit vector of visited nodes).

Here's the first
bit of the output of -Dr execution. I see why the current check doesn't
work in this situation, but am unsure of how one might fix it without
breaking other things

Exactly. I am still investigating. Might take me a few days, but ill get
there.

Yves

How is this coming?

--
perl -Mre=debug -e "/just|another|perl|hacker/"

[p5pRT]

From @khwilliamson

I've added this to the 5.24 blockers list.

Yves, if you don't have the tuits for this, could you post what you got so far?
--
Karl Williamson

[p5pRT]

From @jkeenan

On Tue Oct 13 23​:05​:46 2015, victor@​drawall.cc wrote​:

My bad, my “How to reproduce section” is wrong. When matched against
an empty string, the regex will fail immediately, as expected. The
problems only start when it is applied to a non-empty string, as in :

perl -e '"a"=~/(.(?2))((?<=(?=(?1)).))/'

I’ve just reproduced it on the latest blead
(f83e001).

I’d recommend doing a `ulimit -Sv 1048576` or some such before running
the command, to prevent it from getting out of control.

And here it is at today's blead (on Linux x86_64)​:
#####
$ git show | head -1
commit f24480b

$ ulimit -Sv 1048576 && ./perl -Ilib -e '"a"=~/(.(?2))((?<=(?=(?1)).))/'
Out of memory!
#####

--
James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @demerphq

On 2 March 2016 at 00​:58, James E Keenan via RT
<perlbug-followup@​perl.org> wrote​:

On Tue Oct 13 23​:05​:46 2015, victor@​drawall.cc wrote​:

My bad, my “How to reproduce section” is wrong. When matched against
an empty string, the regex will fail immediately, as expected. The
problems only start when it is applied to a non-empty string, as in :

perl -e '"a"=~/(.(?2))((?<=(?=(?1)).))/'

I’ve just reproduced it on the latest blead
(f83e001).

I’d recommend doing a `ulimit -Sv 1048576` or some such before running
the command, to prevent it from getting out of control.

And here it is at today's blead (on Linux x86_64)​:
#####
$ git show | head -1
commit f24480b

$ ulimit -Sv 1048576 && ./perl -Ilib -e '"a"=~/(.(?2))((?<=(?=(?1)).))/'
Out of memory!

Yeah, I know. This is still on my todo list. All I can say is that I
am now "back" hacking after being away for a while dealing with the
passing of my father and hope to get to it sooner than later.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

[p5pRT]

From @demerphq

On 2 March 2016 at 11​:01, demerphq <demerphq@​gmail.com> wrote​:

On 2 March 2016 at 00​:58, James E Keenan via RT
<perlbug-followup@​perl.org> wrote​:

And here it is at today's blead (on Linux x86_64)​:
#####
$ git show | head -1
commit f24480b

$ ulimit -Sv 1048576 && ./perl -Ilib -e '"a"=~/(.(?2))((?<=(?=(?1)).))/'
Out of memory!

hope to get to it sooner than later.

And this should be fixed now with​:

ba6840f

This might need a doc fix. It is no longer illegal to do "infinite
recursion", instead the match "fails" to match at the given position.

so for instance this​:

/a(?R)?b/

now matches "ab", "aabb", etc.

I am not sure this is the best wording, but now a pattern sub-routine
will fail to match if entered recursively from the same position. Note
this does not apply to EVAL (??{ ... }) style recursion, only GOSUB
style recursion (?&foo).

Eg​:

"foobar"=~/(?&y)(?(DEFINE)(?<x>(?&y)?foo)(?<y>(?&x)?bar))/

matches. However​:

"foobar"=~/(?&y)(?(DEFINE)(?<x>(?&y)foo)(?<y>(?&x)bar))/

does not, and produces output like this​:

floating "foobar" at 0..9223372036854775807 (checking floating) minlen 6
Matching REx "(?&y)(?(DEFINE)(?<x>(?&y)foo)(?<y>(?&x)bar))" against "foobar"
Intuit​: trying to determine minimum start position...
  doing 'check' fbm scan, [0..6] gave 0
  Found floating substr "foobar" at offset 0 (rx_origin now 0)...
  (multiline anchor test skipped)
Intuit​: Successfully guessed​: match at offset 0
  0 <> <foobar> | 1​:GOSUB2[+16​:17] 'y'(4)
  0 <> <foobar> | 17​: OPEN2 'y'(19)
  0 <> <foobar> | 19​: GOSUB1[-11​:8] 'x'(22)
  0 <> <foobar> | 8​: OPEN1 'x'(10)
  0 <> <foobar> | 10​: GOSUB2[+7​:17] 'y'(13)
  pattern left-recursion without
consuming input always fails...
  failed...
Match failed
Freeing REx​: "(?&y)(?(DEFINE)(?<x>(?&y)foo)(?<y>(?&x)bar))"

So I think this ticket can be closed.

cheers,
Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"