split /\A/ works like /^/m, matches embedded newlines

[p5pRT]

Migrated from rt.perl.org#122761 (status was 'resolved')

Searchable as RT122761$

[p5pRT]

From @mauke

perldoc perlrebackslash​:

  \A "\A" only matches at the beginning of the string.

perldoc -f split​:

  Empty leading fields are produced when there are positive-width matches at the beginning of the string; a zero-width match at the beginning of the string does not produce an empty field.

Therefore split /\A/ should return the input string as is. \A can only match once (at offset 0), which (logically speaking) should turn "foo" into ("", "foo"), but because of the special case in split of not producing empty leading fields for zero-width matches at the beginning, we just get "foo" again.

What actually happens​:

$ perl -wE 'say "[$_]" for split /\A/, "foo\nbar\nbaz"'
[foo
]
[bar
]
[baz]

Apparently split thinks /\A/ is the same as /^/m, matching after every embedded newline in the input string. I think this is a bug in split.

The test above was with​:
This is perl 5, version 12, subversion 4 (v5.12.4) built for x86_64-linux

... but an IRC bot running 5.20.0 produces the same results so I assume it's still present in 5.20.

[p5pRT]

From @demerphq

On 11 September 2014 14​:28, l.mai@​web.de <perlbug-followup@​perl.org> wrote​:

# New Ticket Created by l.mai@​web.de
# Please include the string​: [perl #122761]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=122761 >

perldoc perlrebackslash​:

\A "\A" only matches at the beginning of the string.

perldoc -f split​:

Empty leading fields are produced when there are positive-width matches
at the beginning of the string; a zero-width match at the beginning of the
string does not produce an empty field.

Therefore split /\A/ should return the input string as is. \A can only
match once (at offset 0), which (logically speaking) should turn "foo" into
("", "foo"), but because of the special case in split of not producing
empty leading fields for zero-width matches at the beginning, we just get
"foo" again.

What actually happens​:

$ perl -wE 'say "[$_]" for split /\A/, "foo\nbar\nbaz"'
[foo
]
[bar
]
[baz]

Apparently split thinks /\A/ is the same as /^/m, matching after every
embedded newline in the input string. I think this is a bug in split.

The test above was with​:
This is perl 5, version 12, subversion 4 (v5.12.4) built for x86_64-linux

... but an IRC bot running 5.20.0 produces the same results so I assume
it's still present in 5.20.

Yes this is still in blead.

I was party to breaking this in 7bd1e61 in
2007. (7 years to find the bug in the logic!) But the story is, as usual,
much more complicated than that.

This code does NOT use the regex engine for anything other parsing the
pattern. The pattern produces an SBOL END regop sequence, which is the same
as would be produced for /^/, and triggers the RXf_START_ONLY optimisation
case for split.

Part of the problem is that way way way back in the history of Perl,
someone decided that split /^/, $string should behave the same as split
/^/m, $string.

To explain more /^/m produces a MBOL op, "multi-beginning-of-line", and /^/
produces a SBOL op, "single-beginning-of-line".

And split will and has always treated both the same, as an MBOL, when the
pattern was JUST /^/.

Later on in history /\A/ was added as a synonym for /^/, and produces an
SBOL op.

When I upgraded the logic in 7bd1e61 to not look at the pattern
*string*, and instead look at the regop structure instead (a much more
reliable process), and set flags in the pattern which split would then use,
(something required to enable regex engine plug ins to trigger split
optimisations), I inadvertently made it so split /\A/ was the same as split
/^/ which was always the same thing as split /^/m.

So now we have a problem. There is LOADS of code out there that assumes that

split /^/, $string;

is the correct way to split a string into lines.

However it was only true because of the optimisation in split // did not
pay attention to the presence or absence of the /m flag.

So we are now in a jam.

I can do some kind of workaround that makes /\A/ not trigger this
optimisation, but basically split is broken by design for these kind of
cases.

The naive obvious fix would be to document that split // operates with the
m flag set by default, which would explain the unusual behavior of split
/^/, but that would break other patterns.

For instance split /^x/ does not act as though there is an implicit /m flag
set.

perl -le'my $str="foo\nxbar\nxbaz\n"; print ">>$_<<" for split /^x/, $str'

foo
xbar
xbaz
<<
perl -le'my $str="foo\nxbar\nxbaz\n"; print ">>$_<<" for split /^x/m, $str'
foo
<<
bar
<<
baz
<<

Compare with just plain /^/​:

perl -le'my $str="foo\nxbar\nxbaz\n"; print ">>$_<<" for split /^/m, $str'

foo
<<
xbar
<<
xbaz
<<
perl -le'my $str="foo\nxbar\nxbaz\n"; print ">>$_<<" for split /^/, $str'
foo
<<
xbar
<<
xbaz
<<

IOW, split /^/ and split /^/m do the same thing, which they definitely
shouldn't.

Given all this I really cant decide what to do. I *could* change the code
so that SBOL is exempt from this optimization (or perhaps triggered a
different optimisation) but that would break split /^/, on one level I
wouldn't mind, as I could argue it is broken already, but in practice I
think this would break really a lot of code, and at the very least we would
probably want a deprecation cycle. I *could* mess around and figure out a
way to distinguish /^/ from /\A/ even though the two should be identical,
or I could just say "yeah, split doesnt play by the rules, wont-fix".

A simple work around btw would be to write​: /\A|\A/. But that would suck.

I really dont know what to do here. Basically the root of this bug was
created probably in the very early history of Perl.

Another alternative would be to introduce a multiline version of \A, say \L
for this discussion, and then fix split /\A/ and split /\L/ to do the right
thing, and leave split /^/ broken (and document it is broken).

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @Abigail

On Thu, Sep 11, 2014 at 04​:29​:02PM +0200, demerphq wrote​:

[SNIP]

So now we have a problem. There is LOADS of code out there that assumes that

split /^/, $string;

is the correct way to split a string into lines.

However it was only true because of the optimisation in split // did not
pay attention to the presence or absence of the /m flag.

I first thought having "split /^/" mean the same as "split /^/m" was
done intentionally, as it's documented by "perldoc -f split"​:

  A PATTERN of "/^/" is treated as if it were "/^/m", since it
  isn’t much use otherwise.

The third edition of "Programming Perl" documents this behaviour as well --
but not the second edition does not.

But looking at some old commits, this may actually not be the case.

Commit 2cdd06f (Aug 4,
1999/Ilya Zakharevich) makes perl warn on "split /^/",
saying it's usage to mean "split /^/m" is deprecated. Commit
46a8fef (Aug 5, 1999/Paul Marquess)
turns a "warn" into a "Perl_warner" (with the same message).
Then half an hour later, in commit 0e8f60d
(Aug 5, 1999/Jarkko Hietaniemi) the deprecation warning is no longer
on by default.

And then commit 1ec9456
(Jul 25, 2000/M. J. T. Guy) documents the current behaviour. The patch
says "with notes from tchrist and gbarr", and it was the summer of 2000
that people were working on the third edition of "Programming Perl".

I haven't checked the mail archives to see whether that was any discussion.

So we are now in a jam.

I can do some kind of workaround that makes /\A/ not trigger this
optimisation, but basically split is broken by design for these kind of
cases.

The naive obvious fix would be to document that split // operates with the
m flag set by default, which would explain the unusual behavior of split
/^/, but that would break other patterns.

As I said, /^/ implying a /m has already been documented for 14 years.

[SNIP]

Another alternative would be to introduce a multiline version of \A, say \L
for this discussion, and then fix split /\A/ and split /\L/ to do the right
thing, and leave split /^/ broken (and document it is broken).

My suggestion​: leave it as is, and document it. How useful is it to be
able to write​:

  split /\A/ => $foo;

when you could have written

  $foo;

instead?

Fixing it to do the "right thing" seems like a whole lot of work for little
benefit.

Abigail

[p5pRT]

From @mauke

Am Do 11. Sep 2014, 07​:29​:35, demerphq schrieb​:

I really dont know what to do here. Basically the root of this bug was
created probably in the very early history of Perl.

My first idea would be to revert the opcode checking and go back to the pattern source; i.e. do the equivalent of $src eq "^". That would keep backwards compatibility with existing code and the letter of the documentation ("If PATTERN is /^/, ..."). It would also make \A "work" (i.e. do nothing) again.

Then I'd add a deprecation note to the documentation; something like​:

  If PATTERN is /^/, then it is treated as if it used the multiline
  modifier (/^/m). However, this special case is deprecated. Always
  use /^/m in new code.

That leaves the door open to actual deprecation warnings if we decide to remove this feature in a future release.

[p5pRT]

From @demerphq

On 11 September 2014 18​:27, l.mai@​web.de via RT <perlbug-followup@​perl.org>
wrote​:

Am Do 11. Sep 2014, 07​:29​:35, demerphq schrieb​:

I really dont know what to do here. Basically the root of this bug was
created probably in the very early history of Perl.

My first idea would be to revert the opcode checking and go back to the
pattern source; i.e. do the equivalent of $src eq "^". That would keep
backwards compatibility with existing code and the letter of the
documentation ("If PATTERN is /^/, ..."). It would also make \A "work"
(i.e. do nothing) again.

FWIW, I am really against using the raw pattern.

For instance I expect​:

split /(?​:)^/

to be the same as

split /^(?​:)/

to be the same as

split /^/

to be the same as

split /#this splits lines out without capturing the line break
^
#end of comment
/x

I fixed a bunch of issues like this when I redid this code. I am really
against changing back.

Yves

[p5pRT]

From @ikegami

split already says​:

If PATTERN is /^/ , then it is treated as if it used the multiline modifier
<http​://perldoc.perl.org/perlreref.html#OPERATORS> (/^/m ), since it isn't
much use otherwise.

How about

If PATTERN is /^/ or /\A/, then it is treated as if it used the multiline
modifier <http​://perldoc.perl.org/perlreref.html#OPERATORS> (/^/m ), since
it isn't much use otherwise. Preprending (?​:) (e.g. /(?​:)^/) sufficiently
alters the pattern to restore the normal regex behaviour.

On Thu, Sep 11, 2014 at 10​:29 AM, demerphq <demerphq@​gmail.com> wrote​:

On 11 September 2014 14​:28, l.mai@​web.de <perlbug-followup@​perl.org>
wrote​:

# New Ticket Created by l.mai@​web.de
# Please include the string​: [perl #122761]
# in the subject line of all future correspondence about this issue.
# <URL​: https://rt-archive.perl.org/perl5/Ticket/Display.html?id=122761 >

perldoc perlrebackslash​:

\A "\A" only matches at the beginning of the string.

perldoc -f split​:

Empty leading fields are produced when there are positive-width
matches at the beginning of the string; a zero-width match at the beginning
of the string does not produce an empty field.

Therefore split /\A/ should return the input string as is. \A can only
match once (at offset 0), which (logically speaking) should turn "foo" into
("", "foo"), but because of the special case in split of not producing
empty leading fields for zero-width matches at the beginning, we just get
"foo" again.

What actually happens​:

$ perl -wE 'say "[$_]" for split /\A/, "foo\nbar\nbaz"'
[foo
]
[bar
]
[baz]

Apparently split thinks /\A/ is the same as /^/m, matching after every
embedded newline in the input string. I think this is a bug in split.

The test above was with​:
This is perl 5, version 12, subversion 4 (v5.12.4) built for x86_64-linux

... but an IRC bot running 5.20.0 produces the same results so I assume
it's still present in 5.20.

Yes this is still in blead.

I was party to breaking this in 7bd1e61
in 2007. (7 years to find the bug in the logic!) But the story is, as
usual, much more complicated than that.

This code does NOT use the regex engine for anything other parsing the
pattern. The pattern produces an SBOL END regop sequence, which is the same
as would be produced for /^/, and triggers the RXf_START_ONLY optimisation
case for split.

Part of the problem is that way way way back in the history of Perl,
someone decided that split /^/, $string should behave the same as split
/^/m, $string.

To explain more /^/m produces a MBOL op, "multi-beginning-of-line", and
/^/ produces a SBOL op, "single-beginning-of-line".

And split will and has always treated both the same, as an MBOL, when the
pattern was JUST /^/.

Later on in history /\A/ was added as a synonym for /^/, and produces an
SBOL op.

When I upgraded the logic in 7bd1e61 to not look at the pattern
*string*, and instead look at the regop structure instead (a much more
reliable process), and set flags in the pattern which split would then use,
(something required to enable regex engine plug ins to trigger split
optimisations), I inadvertently made it so split /\A/ was the same as split
/^/ which was always the same thing as split /^/m.

So now we have a problem. There is LOADS of code out there that assumes
that

split /^/, $string;

is the correct way to split a string into lines.

However it was only true because of the optimisation in split // did not
pay attention to the presence or absence of the /m flag.

So we are now in a jam.

I can do some kind of workaround that makes /\A/ not trigger this
optimisation, but basically split is broken by design for these kind of
cases.

The naive obvious fix would be to document that split // operates with the
m flag set by default, which would explain the unusual behavior of split
/^/, but that would break other patterns.

For instance split /^x/ does not act as though there is an implicit /m
flag set.

perl -le'my $str="foo\nxbar\nxbaz\n"; print ">>$_<<" for split /^x/, $str'

foo
xbar
xbaz
<<
perl -le'my $str="foo\nxbar\nxbaz\n"; print ">>$_<<" for split /^x/m, $str'
foo
<<
bar
<<
baz
<<

Compare with just plain /^/​:

perl -le'my $str="foo\nxbar\nxbaz\n"; print ">>$_<<" for split /^/m, $str'

foo
<<
xbar
<<
xbaz
<<
perl -le'my $str="foo\nxbar\nxbaz\n"; print ">>$_<<" for split /^/, $str'
foo
<<
xbar
<<
xbaz
<<

IOW, split /^/ and split /^/m do the same thing, which they definitely
shouldn't.

Given all this I really cant decide what to do. I *could* change the code
so that SBOL is exempt from this optimization (or perhaps triggered a
different optimisation) but that would break split /^/, on one level I
wouldn't mind, as I could argue it is broken already, but in practice I
think this would break really a lot of code, and at the very least we would
probably want a deprecation cycle. I *could* mess around and figure out a
way to distinguish /^/ from /\A/ even though the two should be identical,
or I could just say "yeah, split doesnt play by the rules, wont-fix".

A simple work around btw would be to write​: /\A|\A/. But that would suck.

I really dont know what to do here. Basically the root of this bug was
created probably in the very early history of Perl.

Another alternative would be to introduce a multiline version of \A, say
\L for this discussion, and then fix split /\A/ and split /\L/ to do the
right thing, and leave split /^/ broken (and document it is broken).

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

[p5pRT]

From @Abigail

On Thu, Sep 11, 2014 at 09​:27​:36AM -0700, l.mai@​web.de via RT wrote​:

Am Do 11. Sep 2014, 07​:29​:35, demerphq schrieb​:

I really dont know what to do here. Basically the root of this bug was
created probably in the very early history of Perl.

My first idea would be to revert the opcode checking and go back to the pattern source; i.e. do the equivalent of $src eq "^". That would keep backwards compatibility with existing code and the letter of the documentation ("If PATTERN is /^/, ..."). It would also make \A "work" (i.e. do nothing) again.

Then I'd add a deprecation note to the documentation; something like​:

If PATTERN is /^/\, then it is treated as if it used the multiline
modifier \(/^/m\)\. However\, this special case is deprecated\. Always
use /^/m in new code\.

That leaves the door open to actual deprecation warnings if we decide to remove this feature in a future release.

As can been seen in my other post, we did this back in 1999. Then quickly
turned off the warning by default. And then a year later, just documented
the behaviour. It has been documented to work like this for 14 years now,
more than half the life time of Perl.

Considering the uselessness of splitting on just the beginning of the string
(which is effectively a noop), I do not think there's anything significant to
be gained by deprecating this.

Abigail

[p5pRT]

From @demerphq

On 11 September 2014 18​:59, Eric Brine <ikegami@​adaelis.com> wrote​:

split already says​:

If PATTERN is /^/ , then it is treated as if it used the multiline
modifier <http​://perldoc.perl.org/perlreref.html#OPERATORS> (/^/m ),
since it isn't much use otherwise.

How about

If PATTERN is /^/ or /\A/, then it is treated as if it used the multiline
modifier <http​://perldoc.perl.org/perlreref.html#OPERATORS> (/^/m ),
since it isn't much use otherwise. Preprending (?​:) (e.g. /(?​:)^/)
sufficiently alters the pattern to restore the normal regex behaviour.

I really really really hate the idea that prepending (?​:) to the pattern
should change what it does. It is completely counterintuitive. Like

foo((),$thing);

being different from

foo($thing);

And like I said we had a bunch of bug reports along those lines.

The real issue here is that the /^/ implies /m in split thing was not
properly thought out and should never have been done. Why should
C<split /^/, $string> be different from C<split /^x/, $string>?

This is yet another example of how "ooh neat" features, especially in the
regex engine almost *always* cause trouble that is nearly impossible to
resolve after the fact.

Yves

[p5pRT]

From @rjbs

* demerphq <demerphq@​gmail.com> [2014-09-11T10​:29​:02]

I *could* mess around and figure out a way to distinguish /^/ from /\A/ even
though the two should be identical, or I could just say "yeah, split doesnt
play by the rules, wont-fix".

First off​: thanks for this post, which was interesting and useful.

It seems to me that the above is a subset of the below​:

Another alternative would be to introduce a multiline version of \A, say \L
for this discussion, and then fix split /\A/ and split /\L/ to do the right
thing, and leave split /^/ broken (and document it is broken).

That is​: you need to distinguish ^ from \A, whether or not you add \L, for such
a fix. Before I get into anything else​: is that an accurate reading of the
situation?

--
rjbs

[p5pRT]

From @demerphq

On 11 September 2014 19​:54, Ricardo Signes <perl.p5p@​rjbs.manxome.org>
wrote​:

* demerphq <demerphq@​gmail.com> [2014-09-11T10​:29​:02]

I *could* mess around and figure out a way to distinguish /^/ from /\A/
even
though the two should be identical, or I could just say "yeah, split
doesnt
play by the rules, wont-fix".

First off​: thanks for this post, which was interesting and useful.

No problem. Especially as I was indirectly responsible for part of the mess.

It seems to me that the above is a subset of the below​:

Another alternative would be to introduce a multiline version of \A, say
\L
for this discussion, and then fix split /\A/ and split /\L/ to do the
right
thing, and leave split /^/ broken (and document it is broken).

That is​: you need to distinguish ^ from \A, whether or not you add \L, for
such
a fix. Before I get into anything else​: is that an accurate reading of the
situation?

Er, sort of. What you describe is option 2 below.

Thinking about this more I think there are two reasonable options​:

1. document that all patterns to split are compiled under /m by default. At
the same time we would change the optimisation for /^/ to detect MBOL,
which is what is produced by split /^/m, $string. This would then mean that
only /^/ would trigger the optimisation as it would produce a MBOL, and \A
would be fine because it produces an SBOL.

2. use the flag field of the regop to store whether the SBOL comes from \A
or ^, and then only enable the /^/ optimisation when it was /^/.

Personally the more I think about this more i think that 1 is better, even
though it is probably the riskier of the two. Having said that I am
speaking hypothetically, option 1 *might* break something, but I struggle
to think what, whereas option 2 would leave lots of things "broken" that
are already "broken" and would fix this single case only, without breaking
anything else.

Consider what option 1 would result in​:

  split /^/, $string
  split /^x/, $string

would behave the same as far as the ^ operator goes. And it would mean that

split /^/, $string
split /$/, $string
spit /\Z/, $string

would behave similarly (that is match all the beginning or end of lines in
the string). That they dont IMO is pretty wrong. The justification applied
to make split /^/ work IMO applies just as much to split /$/ or split /\z/.

And it would fix the problem with /\A/ behaving like /^/m (which is
uncontroversially wrong).

And when I think about what it would break I struggle to think of
something. Does anything come to mind to anyone else?

Also the other nice thing about option one is it doesnt need an \L
metacharacter, which I proposed only because we would have no way to say "I
really want to split on the beginning of the string". Whereas if we
defaulted split compilation to enable /m then we could turn it off easily​:

split /(?-m​:^)/, $string

would disable the defaut /m flag. The reason I proposed the \L
metacharacter is there is no way to turn off a flag from the outside of the
pattern.

In fact the process of writing this email I have become sufficiently
convinced that option /m is the right thing to do and that I will start
writing the patch now so we can find out if it breaks anything.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

[p5pRT]

From @ap

* demerphq <demerphq@​gmail.com> [2014-09-11 20​:25]​:

Whereas if we defaulted split compilation to enable /m then we could
turn it off easily​:

split /(?-m​:^)/, $string

would disable the defaut /m flag.

Would writing it `split qr/^/, $string` also work? (I would hope yes.)

Regards,
--
Aristotle Pagaltzis // <http​://plasmasturm.org/>

[p5pRT]

From @demerphq

On 11 September 2014 22​:45, Aristotle Pagaltzis <pagaltzis@​gmx.de> wrote​:

* demerphq <demerphq@​gmail.com> [2014-09-11 20​:25]​:

Whereas if we defaulted split compilation to enable /m then we could
turn it off easily​:

split /(?-m​:^)/, $string

would disable the defaut /m flag.

Would writing it `split qr/^/, $string` also work? (I would hope yes.)

No, when Karl changed qr/^/ to reduce down to (?^​:^) he changed the
semantics of such a case so we would not disable the /m flag, and this
would behave just the same as split /^/, $_. I think anyway.

IOW, (?^​:^) means "match /^/ under whatever rules the pattern is compiled
in".

In the older perls it would turn into (?-msix​:^) and then yes I think it
would have behaved as you expect.

Win-some, lose-some.

cheers,
Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

[p5pRT]

From @ap

* demerphq <demerphq@​gmail.com> [2014-09-11 22​:55]​:

No, when Karl changed qr/^/ to reduce down to (?^​:^) he changed the
semantics of such a case so we would not disable the /m flag, and this
would behave just the same as split /^/, $_.

It does.

I am assuming that this special compilation context applies at the time
that split compiles the pattern, which implies that if split were made
to not stringify qr objects, then it would not apply to them. Correct?

If so – is that doable with reasonable effort?

It would go some ways toward regularising split’s behaviour further.

Regards,
--
Aristotle Pagaltzis // <http​://plasmasturm.org/>

[p5pRT]

From @Abigail

On Thu, Sep 11, 2014 at 08​:20​:28PM +0200, demerphq wrote​:

Thinking about this more I think there are two reasonable options​:

1. document that all patterns to split are compiled under /m by default.

To do that, you would first have to change the behavior of split, as
it currently does *NOT* do this. Only for /^/. Witness​:

  $ perl -E 'say "[$_]" for split /^a/m => "foo\nabar\nabaz"'
  [foo
  ]
  [bar
  ]
  [baz]

  $ perl -E 'say "[$_]" for split /^a/ => "foo\nabar\nabaz"'
  [foo
  abar
  abaz]

Abigail