Bleadperl v5.21.4-88-ga51d618 breaks DCONWAY/Regexp-Grammars-1.036.tar.gz #14129

p5pRT · 2014-10-03T06:19:01Z

Migrated from rt.perl.org#122890 (status was 'resolved')

Searchable as RT122890$

p5pRT · 2014-10-03T06:19:01Z

From @andk

git bisect

commit a51d618
Author: Yves Orton <demerphq@gmail.com>
Date: Fri Sep 19 19:57:34 2014 +0200

rt 122283 - do not recurse into GOSUB/GOSTART when not SCF_DO_SUBSTR

sample fail report

http://www.cpantesters.org/cpan/report/98605662-4578-11e4-966f-9b262c6f0924

perl -V

Summary of my perl5 (revision 5 version 21 subversion 5) configuration:
Commit id: a83034f
Platform:
osname=linux, osvers=3.16-1-amd64, archname=x86_64-linux-thread-multi-ld
uname='linux k83 3.16-1-amd64 #1 smp debian 3.16.2-2 (2014-09-08) x86_64 gnulinux '
config_args='-Dprefix=/home/sand/src/perl/repoperls/installed-perls/perl/v5.21.4-101-ga83034f/a2da -Dmyhostname=k83 -Dinstallusrbinperl=n -Uversiononly -Dusedevel -des -Ui_db -Duseithreads -Duselongdouble -DDEBUGGING=-g'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
use64bitint=define, use64bitall=define, uselongdouble=define
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2',
optimize='-O2 -g',
cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
ccversion='', gccversion='4.9.1', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
ivtype='long', ivsize=8, nvtype='long double', nvsize=16, Off_t='off_t', lseeksize=8
alignbytes=16, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -fstack-protector-strong -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/4.9/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc -lgdbm_compat
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
libc=libc-2.19.so, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.19'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib -fstack-protector-strong'

Characteristics of this binary (from libperl):
Compile-time options: HAS_TIMES MULTIPLICITY PERLIO_LAYERS
PERL_DONT_CREATE_GVSV
PERL_HASH_FUNC_ONE_AT_A_TIME_HARD
PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP
PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV
PERL_USE_DEVEL USE_64_BIT_ALL USE_64_BIT_INT
USE_ITHREADS USE_LARGE_FILES USE_LOCALE
USE_LOCALE_COLLATE USE_LOCALE_CTYPE
USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_LONG_DOUBLE
USE_PERLIO USE_PERL_ATOF USE_REENTRANT_API
Built under linux
Compiled at Sep 25 2014 22:31:36
%ENV:
PERL5LIB=""
PERL5OPT=""
PERL5_CPANPLUS_IS_RUNNING="21904"
PERL5_CPAN_IS_RUNNING="21904"
PERL_MM_USE_DEFAULT="1"
@INC:
/home/sand/src/perl/repoperls/installed-perls/perl/v5.21.4-101-ga83034f/a2da/lib/site_perl/5.21.5/x86_64-linux-thread-multi-ld
/home/sand/src/perl/repoperls/installed-perls/perl/v5.21.4-101-ga83034f/a2da/lib/site_perl/5.21.5
/home/sand/src/perl/repoperls/installed-perls/perl/v5.21.4-101-ga83034f/a2da/lib/5.21.5/x86_64-linux-thread-multi-ld
/home/sand/src/perl/repoperls/installed-perls/perl/v5.21.4-101-ga83034f/a2da/lib/5.21.5
.
--
andreas

p5pRT · 2014-10-04T05:03:48Z

From @cpansprout

On Thu Oct 02 23:19:01 2014, andreas.koenig.7os6VVqR@franz.ak.mind.de wrote:

git bisect
----------
commit a51d618
Author: Yves Orton <demerphq@gmail.com>
Date: Fri Sep 19 19:57:34 2014 +0200

rt 122283 - do not recurse into GOSUB/GOSTART when not SCF_DO_SUBSTR

sample fail report
------------------
http://www.cpantesters.org/cpan/report/98605662-4578-11e4-966f-
9b262c6f0924

Reduced case:

print scalar '7% solution' =~
m/(?: (?&solution) | % ) \Z
(?(DEFINE) (?<solution>7\%\ solution) )
/msx, "\n";

Output in 5.21.4:
1

Output in blead is an empty line.

--

Father Chrysostomos

p5pRT · 2014-10-04T05:03:48Z

The RT System itself - Status changed from 'new' to 'open'

p5pRT · 2014-10-05T01:58:37Z

From @cpansprout

On Fri Oct 03 22:03:48 2014, sprout wrote:

On Thu Oct 02 23:19:01 2014, andreas.koenig.7os6VVqR@franz.ak.mind.de wrote:

git bisect
----------
commit a51d618
Author: Yves Orton <demerphq@gmail.com>
Date: Fri Sep 19 19:57:34 2014 +0200

rt 122283 - do not recurse into GOSUB/GOSTART when not SCF_DO_SUBSTR

sample fail report
------------------
http://www.cpantesters.org/cpan/report/98605662-4578-11e4-966f-
9b262c6f0924

Reduced case:

print scalar '7% solution' =~
m/(?: (?&solution) | % ) \Z
(?(DEFINE) (?<solution>7\%\ solution) )
/msx, "\n";

Output in 5.21.4:
1

Output in blead is an empty line.

More info: The anchored-end optimisation determines that the pattern cannot start earlier than position 10 (1 character before the end of the string. Apparently it sees (?&solution) as having zero length, instead of potentially infinite length. If study_chunk (or whatever is responsible) is not going to follow (?&...) recursively, it at least needs to note that max-length optimisations need to be skipped. How does one go about that?

Compiling REx "(?: (?&solution) | %% ) \Z%n (?(DEFINE) (?<solution>7\%%\"...
rarest char
at 0
Final program:
1: BRANCH (5)
2: GOSUB1[+12] (9)
5: BRANCH (FAIL)
6: EXACT <%> (9)
8: TAIL (9)
9: SEOL (10)
10: DEFINEP (12)
12: IFTHEN (25)
14: OPEN1 'solution' (16)
16: EXACT <7% solution> (20)
20: CLOSE1 'solution' (25)
22: LONGJMP (24)
24: TAIL (25)
25: END (0)
floating ""$ at 0..1 (checking floating) minlen 0
Enabling $` $& $' support (0x7).

EXECUTING...

Matching REx "(?: (?&solution) | %% ) \Z%n (?(DEFINE) (?<solution>7\%%\"... against "7%% solution"
Intuit: trying to determine minimum start position...
Found floating substr ""$ at offset 11...
(multiline anchor test skipped)
try at offset...
Intuit: Successfully guessed: match at offset 10
10 <7%% solutio> <n> | 1:BRANCH(5)

etc.

--

Father Chrysostomos

p5pRT · 2014-10-05T09:48:32Z

From @demerphq

On 5 October 2014 03:58, Father Chrysostomos via RT <
perlbug-followup@perl.org> wrote:

On Fri Oct 03 22:03:48 2014, sprout wrote:

On Thu Oct 02 23:19:01 2014, andreas.koenig.7os6VVqR@franz.ak.mind.de
wrote:

git bisect
----------
commit a51d618
Author: Yves Orton <demerphq@gmail.com>
Date: Fri Sep 19 19:57:34 2014 +0200

rt 122283 - do not recurse into GOSUB/GOSTART when not SCF_DO_SUBSTR

sample fail report
------------------
http://www.cpantesters.org/cpan/report/98605662-4578-11e4-966f-
9b262c6f0924

Reduced case:

print scalar '7% solution' =~
m/(?: (?&solution) | % ) \Z
(?(DEFINE) (?<solution>7\%\ solution) )
/msx, "\n";

Output in 5.21.4:
1

Output in blead is an empty line.

More info: The anchored-end optimisation determines that the pattern
cannot start earlier than position 10 (1 character before the end of the
string. Apparently it sees (?&solution) as having zero length, instead of
potentially infinite length. If study_chunk (or whatever is responsible)
is not going to follow (?&...) recursively, it at least needs to note that
max-length optimisations need to be skipped. How does one go about that?

I don't think it is that simple. There are a bunch of subtle behaviors
interacting here in a suboptimal way.

I will try to find time for this this week. Your reduction to simpler case
makes my life a lot easier, thanks.

For now please leave this to me.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

p5pRT · 2014-10-13T06:41:03Z

From @demerphq

On 5 October 2014 11:47, demerphq <demerphq@gmail.com> wrote:

On 5 October 2014 03:58, Father Chrysostomos via RT <
perlbug-followup@perl.org> wrote:

On Fri Oct 03 22:03:48 2014, sprout wrote:

On Thu Oct 02 23:19:01 2014, andreas.koenig.7os6VVqR@franz.ak.mind.de
wrote:

git bisect
----------
commit a51d618
Author: Yves Orton <demerphq@gmail.com>
Date: Fri Sep 19 19:57:34 2014 +0200

rt 122283 - do not recurse into GOSUB/GOSTART when not SCF_DO_SUBSTR

sample fail report
------------------
http://www.cpantesters.org/cpan/report/98605662-4578-11e4-966f-
9b262c6f0924

Reduced case:

print scalar '7% solution' =~
m/(?: (?&solution) | % ) \Z
(?(DEFINE) (?<solution>7\%\ solution) )
/msx, "\n";

Output in 5.21.4:
1

Output in blead is an empty line.

More info: The anchored-end optimisation determines that the pattern
cannot start earlier than position 10 (1 character before the end of the
string. Apparently it sees (?&solution) as having zero length, instead of
potentially infinite length. If study_chunk (or whatever is responsible)
is not going to follow (?&...) recursively, it at least needs to note that
max-length optimisations need to be skipped. How does one go about that?

I don't think it is that simple. There are a bunch of subtle behaviors
interacting here in a suboptimal way.

I will try to find time for this this week. Your reduction to simpler case
makes my life a lot easier, thanks.

For now please leave this to me.

For the record I put some hours into this over the weekend. No final result
yet, but I have fixed the memory leaks. The rt 122283 pattern now runs to
finish in 372 minutes. :-)

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

p5pRT · 2014-10-19T07:52:21Z

From @andk

also affected:
SPROUT/JE-0.065.tar.gz
http://www.cpantesters.org/cpan/report/785cae16-54f9-11e4-b101-b4d49ed67017
--
andreas

p5pRT · 2014-10-20T06:45:50Z

From @demerphq

On 19 October 2014 09:51, Andreas Koenig <
andreas.koenig.7os6VVqR@franz.ak.mind.de> wrote:

also affected:
SPROUT/JE-0.065.tar.gz
http://www.cpantesters.org/cpan/report/785cae16-54f9-11e4-b101-b4d49ed67017

Last night I pushed a patch sequence which hopefully fixes this and RT #
122283.

If that fixes these modules as well then I consider this issue finally
properly closed.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

p5pRT · 2014-10-20T18:17:08Z

From @andk

On Mon, 20 Oct 2014 08:45:15 +0200, demerphq <demerphq@gmail.com> said:

> On 19 October 2014 09:51, Andreas Koenig <
> andreas.koenig.7os6VVqR@franz.ak.mind.de> wrote:

> also affected:
> SPROUT/JE-0.065.tar.gz
> http://www.cpantesters.org/cpan/report/785cae16-54f9-11e4-b101-b4d49ed67017

> Last night I pushed a patch sequence which hopefully fixes this and RT #
> 122283.

> If that fixes these modules as well then I consider this issue finally properly
> closed.

With v5.21.4-553-gf6f22b1:

DCONWAY/Regexp-Grammars-1.036.tar.gz
./Build test -- OK

SPROUT/JE-0.065.tar.gz
/usr/bin/make test -- OK

Thank You && Cheers!
--
andreas

p5pRT · 2014-10-20T20:37:47Z

From @demerphq

On 20 October 2014 20:16, Andreas Koenig <
andreas.koenig.7os6VVqR@franz.ak.mind.de> wrote:

On Mon, 20 Oct 2014 08:45:15 +0200, demerphq <demerphq@gmail.com>
said:

On 19 October 2014 09:51, Andreas Koenig <
andreas.koenig.7os6VVqR@franz.ak.mind.de> wrote:
also affected&#8203;:
SPROUT/JE\-0\.065\.tar\.gz
http://www.cpantesters.org/cpan/report/785cae16-54f9-11e4-b101-b4d49ed67017

Last night I pushed a patch sequence which hopefully fixes this and RT
#
122283.

If that fixes these modules as well then I consider this issue finally
properly
closed.

With v5.21.4-553-gf6f22b1:

DCONWAY/Regexp-Grammars-1.036.tar.gz
./Build test -- OK

SPROUT/JE-0.065.tar.gz
/usr/bin/make test -- OK

Thank You && Cheers!

And thank you for your CPAN reports. :-)

cheers,
Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

p5pRT · 2014-10-20T20:52:37Z

@cpansprout - Status changed from 'open' to 'resolved'

p5pRT closed this Oct 20, 2014

p5pRT added the Severity Low label Oct 19, 2019

Perl / perl5

Bleadperl v5.21.4-88-ga51d618 breaks DCONWAY/Regexp-Grammars-1.036.tar.gz #14129

Bleadperl v5.21.4-88-ga51d618 breaks DCONWAY/Regexp-Grammars-1.036.tar.gz #14129

p5pRT commented Oct 3, 2014

p5pRT commented Oct 3, 2014

p5pRT commented Oct 4, 2014

p5pRT commented Oct 4, 2014

p5pRT commented Oct 5, 2014

p5pRT commented Oct 5, 2014

p5pRT commented Oct 13, 2014

p5pRT commented Oct 19, 2014

p5pRT commented Oct 20, 2014

p5pRT commented Oct 20, 2014

p5pRT commented Oct 20, 2014

p5pRT commented Oct 20, 2014

Perl / perl5

Bleadperl v5.21.4-88-ga51d618 breaks DCONWAY/Regexp-Grammars-1.036.tar.gz #14129

Bleadperl v5.21.4-88-ga51d618 breaks DCONWAY/Regexp-Grammars-1.036.tar.gz #14129

Comments

p5pRT commented Oct 3, 2014

p5pRT commented Oct 3, 2014

From @andk

p5pRT commented Oct 4, 2014

From @cpansprout

p5pRT commented Oct 4, 2014

p5pRT commented Oct 5, 2014

From @cpansprout

p5pRT commented Oct 5, 2014

From @demerphq

p5pRT commented Oct 13, 2014

From @demerphq

p5pRT commented Oct 19, 2014

From @andk

p5pRT commented Oct 20, 2014

From @demerphq

p5pRT commented Oct 20, 2014

From @andk

p5pRT commented Oct 20, 2014

From @demerphq

p5pRT commented Oct 20, 2014