Bleadperl v5.21.4-88-ga51d618 breaks DCONWAY/Regexp-Grammars-1.036.tar.gz

[p5pRT]

Migrated from rt.perl.org#122890 (status was 'resolved')

Searchable as RT122890$

[p5pRT]

From @andk

git bisect

---

commit a51d618
Author​: Yves Orton <demerphq@​gmail.com>
Date​: Fri Sep 19 19​:57​:34 2014 +0200

  rt 122283 - do not recurse into GOSUB/GOSTART when not SCF_DO_SUBSTR

sample fail report

---

http​://www.cpantesters.org/cpan/report/98605662-4578-11e4-966f-9b262c6f0924

perl -V

---

Summary of my perl5 (revision 5 version 21 subversion 5) configuration​:
  Commit id​: a83034f
  Platform​:
  osname=linux, osvers=3.16-1-amd64, archname=x86_64-linux-thread-multi-ld
  uname='linux k83 3.16-1-amd64 #1 smp debian 3.16.2-2 (2014-09-08) x86_64 gnulinux '
  config_args='-Dprefix=/home/sand/src/perl/repoperls/installed-perls/perl/v5.21.4-101-ga83034f/a2da -Dmyhostname=k83 -Dinstallusrbinperl=n -Uversiononly -Dusedevel -des -Ui_db -Duseithreads -Duselongdouble -DDEBUGGING=-g'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=define, usemultiplicity=define
  use64bitint=define, use64bitall=define, uselongdouble=define
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2',
  optimize='-O2 -g',
  cppflags='-D_REENTRANT -D_GNU_SOURCE -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
  ccversion='', gccversion='4.9.1', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
  ivtype='long', ivsize=8, nvtype='long double', nvsize=16, Off_t='off_t', lseeksize=8
  alignbytes=16, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -fstack-protector-strong -L/usr/local/lib'
  libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/4.9/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
  libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc -lgdbm_compat
  perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
  libc=libc-2.19.so, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version='2.19'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib -fstack-protector-strong'

Characteristics of this binary (from libperl)​:
  Compile-time options​: HAS_TIMES MULTIPLICITY PERLIO_LAYERS
  PERL_DONT_CREATE_GVSV
  PERL_HASH_FUNC_ONE_AT_A_TIME_HARD
  PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP
  PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV
  PERL_USE_DEVEL USE_64_BIT_ALL USE_64_BIT_INT
  USE_ITHREADS USE_LARGE_FILES USE_LOCALE
  USE_LOCALE_COLLATE USE_LOCALE_CTYPE
  USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_LONG_DOUBLE
  USE_PERLIO USE_PERL_ATOF USE_REENTRANT_API
  Built under linux
  Compiled at Sep 25 2014 22​:31​:36
  %ENV​:
  PERL5LIB=""
  PERL5OPT=""
  PERL5_CPANPLUS_IS_RUNNING="21904"
  PERL5_CPAN_IS_RUNNING="21904"
  PERL_MM_USE_DEFAULT="1"
  @​INC​:
  /home/sand/src/perl/repoperls/installed-perls/perl/v5.21.4-101-ga83034f/a2da/lib/site_perl/5.21.5/x86_64-linux-thread-multi-ld
  /home/sand/src/perl/repoperls/installed-perls/perl/v5.21.4-101-ga83034f/a2da/lib/site_perl/5.21.5
  /home/sand/src/perl/repoperls/installed-perls/perl/v5.21.4-101-ga83034f/a2da/lib/5.21.5/x86_64-linux-thread-multi-ld
  /home/sand/src/perl/repoperls/installed-perls/perl/v5.21.4-101-ga83034f/a2da/lib/5.21.5
  .
--
andreas

[p5pRT]

From @cpansprout

On Thu Oct 02 23​:19​:01 2014, andreas.koenig.7os6VVqR@​franz.ak.mind.de wrote​:

git bisect
----------
commit a51d618
Author​: Yves Orton <demerphq@​gmail.com>
Date​: Fri Sep 19 19​:57​:34 2014 +0200

rt 122283 - do not recurse into GOSUB/GOSTART when not SCF_DO_SUBSTR

sample fail report
------------------
http​://www.cpantesters.org/cpan/report/98605662-4578-11e4-966f-
9b262c6f0924

Reduced case​:

print scalar '7% solution' =~
  m/(?​: (?&solution) | % ) \Z
  (?(DEFINE) (?<solution>7\%\ solution) )
  /msx, "\n";

Output in 5.21.4​:
1

Output in blead is an empty line.

--

Father Chrysostomos

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @cpansprout

On Fri Oct 03 22​:03​:48 2014, sprout wrote​:

On Thu Oct 02 23​:19​:01 2014, andreas.koenig.7os6VVqR@​franz.ak.mind.de wrote​:

git bisect
----------
commit a51d618
Author​: Yves Orton <demerphq@​gmail.com>
Date​: Fri Sep 19 19​:57​:34 2014 +0200

rt 122283 - do not recurse into GOSUB/GOSTART when not SCF_DO_SUBSTR

sample fail report
------------------
http​://www.cpantesters.org/cpan/report/98605662-4578-11e4-966f-
9b262c6f0924

Reduced case​:

print scalar '7% solution' =~
m/(?​: (?&solution) | % ) \Z
(?(DEFINE) (?<solution>7\%\ solution) )
/msx, "\n";

Output in 5.21.4​:
1

Output in blead is an empty line.

More info​: The anchored-end optimisation determines that the pattern cannot start earlier than position 10 (1 character before the end of the string. Apparently it sees (?&solution) as having zero length, instead of potentially infinite length. If study_chunk (or whatever is responsible) is not going to follow (?&...) recursively, it at least needs to note that max-length optimisations need to be skipped. How does one go about that?

Compiling REx "(?​: (?&solution) | %% ) \Z%n (?(DEFINE) (?<solution>7\%%\"...
rarest char
at 0
Final program​:
  1​: BRANCH (5)
  2​: GOSUB1[+12] (9)
  5​: BRANCH (FAIL)
  6​: EXACT <%> (9)
  8​: TAIL (9)
  9​: SEOL (10)
  10​: DEFINEP (12)
  12​: IFTHEN (25)
  14​: OPEN1 'solution' (16)
  16​: EXACT <7% solution> (20)
  20​: CLOSE1 'solution' (25)
  22​: LONGJMP (24)
  24​: TAIL (25)
  25​: END (0)
floating ""$ at 0..1 (checking floating) minlen 0
Enabling $` $&amp; $' support (0x7).

EXECUTING...

Matching REx "(?​: (?&solution) | %% ) \Z%n (?(DEFINE) (?<solution>7\%%\"... against "7%% solution"
Intuit​: trying to determine minimum start position...
  Found floating substr ""$ at offset 11...
  (multiline anchor test skipped)
  try at offset...
Intuit​: Successfully guessed​: match at offset 10
  10 <7%% solutio> <n> | 1​:BRANCH(5)

etc.

--

Father Chrysostomos

[p5pRT]

From @demerphq

On 5 October 2014 03​:58, Father Chrysostomos via RT <
perlbug-followup@​perl.org> wrote​:

On Fri Oct 03 22​:03​:48 2014, sprout wrote​:

On Thu Oct 02 23​:19​:01 2014, andreas.koenig.7os6VVqR@​franz.ak.mind.de
wrote​:

git bisect
----------
commit a51d618
Author​: Yves Orton <demerphq@​gmail.com>
Date​: Fri Sep 19 19​:57​:34 2014 +0200

rt 122283 - do not recurse into GOSUB/GOSTART when not SCF_DO_SUBSTR

sample fail report
------------------
http​://www.cpantesters.org/cpan/report/98605662-4578-11e4-966f-
9b262c6f0924

Reduced case​:

print scalar '7% solution' =~
m/(?​: (?&solution) | % ) \Z
(?(DEFINE) (?<solution>7\%\ solution) )
/msx, "\n";

Output in 5.21.4​:
1

Output in blead is an empty line.

More info​: The anchored-end optimisation determines that the pattern
cannot start earlier than position 10 (1 character before the end of the
string. Apparently it sees (?&solution) as having zero length, instead of
potentially infinite length. If study_chunk (or whatever is responsible)
is not going to follow (?&...) recursively, it at least needs to note that
max-length optimisations need to be skipped. How does one go about that?

I don't think it is that simple. There are a bunch of subtle behaviors
interacting here in a suboptimal way.

I will try to find time for this this week. Your reduction to simpler case
makes my life a lot easier, thanks.

For now please leave this to me.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

[p5pRT]

From @demerphq

On 5 October 2014 11​:47, demerphq <demerphq@​gmail.com> wrote​:

On 5 October 2014 03​:58, Father Chrysostomos via RT <
perlbug-followup@​perl.org> wrote​:

On Fri Oct 03 22​:03​:48 2014, sprout wrote​:

On Thu Oct 02 23​:19​:01 2014, andreas.koenig.7os6VVqR@​franz.ak.mind.de
wrote​:

git bisect
----------
commit a51d618
Author​: Yves Orton <demerphq@​gmail.com>
Date​: Fri Sep 19 19​:57​:34 2014 +0200

rt 122283 - do not recurse into GOSUB/GOSTART when not SCF_DO_SUBSTR

sample fail report
------------------
http​://www.cpantesters.org/cpan/report/98605662-4578-11e4-966f-
9b262c6f0924

Reduced case​:

print scalar '7% solution' =~
m/(?​: (?&solution) | % ) \Z
(?(DEFINE) (?<solution>7\%\ solution) )
/msx, "\n";

Output in 5.21.4​:
1

Output in blead is an empty line.

More info​: The anchored-end optimisation determines that the pattern
cannot start earlier than position 10 (1 character before the end of the
string. Apparently it sees (?&solution) as having zero length, instead of
potentially infinite length. If study_chunk (or whatever is responsible)
is not going to follow (?&...) recursively, it at least needs to note that
max-length optimisations need to be skipped. How does one go about that?

I don't think it is that simple. There are a bunch of subtle behaviors
interacting here in a suboptimal way.

I will try to find time for this this week. Your reduction to simpler case
makes my life a lot easier, thanks.

For now please leave this to me.

For the record I put some hours into this over the weekend. No final result
yet, but I have fixed the memory leaks. The rt 122283 pattern now runs to
finish in 372 minutes. :-)

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

[p5pRT]

From @andk

also affected​:
SPROUT/JE-0.065.tar.gz
http​://www.cpantesters.org/cpan/report/785cae16-54f9-11e4-b101-b4d49ed67017
--
andreas

[p5pRT]

From @demerphq

On 19 October 2014 09​:51, Andreas Koenig <
andreas.koenig.7os6VVqR@​franz.ak.mind.de> wrote​:

also affected​:
SPROUT/JE-0.065.tar.gz
http​://www.cpantesters.org/cpan/report/785cae16-54f9-11e4-b101-b4d49ed67017

Last night I pushed a patch sequence which hopefully fixes this and RT #
122283.

If that fixes these modules as well then I consider this issue finally
properly closed.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

[p5pRT]

From @andk

On Mon, 20 Oct 2014 08​:45​:15 +0200, demerphq <demerphq@​gmail.com> said​:

  > On 19 October 2014 09​:51, Andreas Koenig <
  > andreas.koenig.7os6VVqR@​franz.ak.mind.de> wrote​:

  > also affected​:
  > SPROUT/JE-0.065.tar.gz
  > http​://www.cpantesters.org/cpan/report/785cae16-54f9-11e4-b101-b4d49ed67017

  > Last night I pushed a patch sequence which hopefully fixes this and RT #
  > 122283.

  > If that fixes these modules as well then I consider this issue finally properly
  > closed.

With v5.21.4-553-gf6f22b1​:

  DCONWAY/Regexp-Grammars-1.036.tar.gz
  ./Build test -- OK

  SPROUT/JE-0.065.tar.gz
  /usr/bin/make test -- OK

Thank You && Cheers!
--
andreas

[p5pRT]

From @demerphq

On 20 October 2014 20​:16, Andreas Koenig <
andreas.koenig.7os6VVqR@​franz.ak.mind.de> wrote​:

On Mon, 20 Oct 2014 08​:45​:15 +0200, demerphq <demerphq@​gmail.com>
said​:

On 19 October 2014 09​:51, Andreas Koenig <
andreas.koenig.7os6VVqR@​franz.ak.mind.de> wrote​:

also affected​:
SPROUT/JE\-0\.065\.tar\.gz

http​://www.cpantesters.org/cpan/report/785cae16-54f9-11e4-b101-b4d49ed67017

Last night I pushed a patch sequence which hopefully fixes this and RT
#
122283.

If that fixes these modules as well then I consider this issue finally
properly
closed.

With v5.21.4-553-gf6f22b1​:

DCONWAY/Regexp-Grammars-1.036.tar.gz
./Build test -- OK

SPROUT/JE-0.065.tar.gz
/usr/bin/make test -- OK

Thank You && Cheers!

And thank you for your CPAN reports. :-)

cheers,
Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

[p5pRT]

@cpansprout - Status changed from 'open' to 'resolved'