chomp() can be confusing

[p5pRT]

Migrated from rt.perl.org#1807 (status was 'rejected')

Searchable as RT1807$

[p5pRT]

From Ben_Tilly@trepp.com

Is there any possibility of having chomp() be modified to recognize \n, \r,
and \r\n as line-endings to chomp? A source of nasty confusion for people
working in a cross-platform environment is when the same exact script gives
very different results on the same exact file depending on whether you are
running under *nix or Windows. (Particularly an issue with Samba because
people wind up reading under one system files created under the other.)

Yes, the current behaviour works as documented. But it leads to code not
doing what people expect, and a confused person can easily spend several
hours confused...

Cheers,
Ben

[p5pRT]

From Ben_Tilly@trepp.com

Is there any possibility of having Perl's chomp() command be modified to
recognize \n, \r, and \r\n as line-endings to chomp? A source of nasty
confusion for people working in a cross-platform environment is when
identical Perl scripts give very different results on the same exact file
depending on whether you are running under *nix or Windows. (Particularly
an issue with Samba because people wind up reading under one system files
created under the other.)

Yes, the current behaviour works as documented. But it leads to code not
doing what people expect, and in many cases a confused person will spend
several hours confused...

Cheers,
Ben

[p5pRT]

From [Unknown Contact. See original ticket]

In message <OF47CC4F38.CF9B333E-ON8525682E.0054550B@​trepp.com>,
  Ben_Tilly@​trepp.com writes​:

: Is there any possibility of having Perl's chomp() command be modified to
: recognize \n, \r, and \r\n as line-endings to chomp? A source of nasty
: confusion for people working in a cross-platform environment is when
: identical Perl scripts give very different results on the same exact file
: depending on whether you are running under *nix or Windows. (Particularly
: an issue with Samba because people wind up reading under one system files
: created under the other.)

Have a look at <URL​:http​://language.perl.com/ppt/src/nlcvt/nlcvt> for
an example of how to do what you want (or at least something similar).

Greg

[p5pRT]

From [Unknown Contact. See original ticket]

In message <OF47CC4F38.CF9B333E-ON8525682E.0054550B@​trepp.com>,
Ben_Tilly@​trepp.com writes​:

: Is there any possibility of having Perl's chomp() command be modified
to
: recognize \n, \r, and \r\n as line-endings to chomp? A source of nasty
: confusion for people working in a cross-platform environment is when
: identical Perl scripts give very different results on the same exact
file
: depending on whether you are running under *nix or Windows.
(Particularly
: an issue with Samba because people wind up reading under one system
files
: created under the other.)

Have a look at <URL​:http​://language.perl.com/ppt/src/nlcvt/nlcvt> for
an example of how to do what you want (or at least something similar).

When I said, "I talked to" I meant that. I don't need the pointer - I know
how to handle it. But I wind up answering questions from people who do
not. In my experience most of them use chomp() so the confusion is
preventable. (After all you expect chomp() to get rid of line endings,
right?)

Cheers,
Ben

[p5pRT]

From [Unknown Contact. See original ticket]

When I said, "I talked to" I meant that. I don't need the pointer - I know
how to handle it. But I wind up answering questions from people who do
not. In my experience most of them use chomp() so the confusion is
preventable. (After all you expect chomp() to get rid of line endings,
right?)

I expect chomp() to remove one and only one terminating instance of the
precise string to which $/ has been set; no more, no less. What were
you expecting?

--tom

[p5pRT]

From @pudge

At 10.22 -0500 1999.11.19, Ben_Tilly@​trepp.com wrote​:

I have just talked to one too many people who have been bitten by this...

Is there any possibility of having Perl's chomp() command be modified to
recognize \n, \r, and \r\n as line-endings to chomp? A source of nasty
confusion for people working in a cross-platform environment is when
identical Perl scripts give very different results on the same exact file
depending on whether you are running under *nix or Windows. (Particularly
an issue with Samba because people wind up reading under one system files
created under the other.)

Yes, the current behaviour works as documented. But it leads to code not
doing what people expect, and in many cases a confused person will spend
several hours confused...

Well, these sam people will also have a problem with readline and <>. And
I certainly don't want behavior where readline and chomp treat different
things as record separators.

I would like to see, perhaps, a regex IRS, so you could do​:

  $/ = qr/(?​:\015\012?|\012)/;

or whatever. Of course, that is flawed, in that it won't catch the special
(usually broken) case of a file having CR, LF, or CRLF mixed in the same
file. Oh well.

Another solution would involve per-filehandle IRS, where you could call a
function (say, textmode()) that would inspect the filehandle and set the
IRS appropriately for that filehandle. This is more subject to failure for
sockets, though, because it would involve reading, looking at the data, and
then seeking back to the beginning.

I have a prototype of something that tied filehandles to do this, but it
fails with sockets, and doesn't do anything for chomp() anyway (and didn't
work great anyway because of some flaws in tied filehandles and prototypes
... I was using 5.004, I don't know if the flaws have been fixed or
whatnot).

--
Chris Nandor mailto​:pudge@​pobox.com http​://pudge.net/
%PGPKey = ('B76E72AD', [1024, '0824090B CE73CA10 1FF77F13 8180B6B6'])

[p5pRT]

From [Unknown Contact. See original ticket]

Ben_Tilly@​trepp.com wrote

Is there any possibility of having Perl's chomp() command be modified to
recognize \n, \r, and \r\n as line-endings to chomp?

I hope not. chomp() should match the string in $?, no more or less.

Your problem is not with chomp(). Rather it is with the I/O subsystem.
If you are reading a file as a newline-terminated text file, then
what your Perl code should see is "\n" and nothing else.

You can achieve this with tied filehandles, but I understand that
isn't what you're looking for.

I think the "right" way of doing this is by providing some sort of
filter apparatus on files. Things of this sort were discussed in
the context of unicode. I don't recall where (if anywhere) that
ended.

Mike Guy

[p5pRT]

From [Unknown Contact. See original ticket]

I think the "right" way of doing this is by providing some sort of
filter apparatus on files.

Well, that makes two of us.

Things of this sort were discussed in
the context of unicode. I don't recall where (if anywhere) that
ended.

Check mjd's summaries?

--tom

[p5pRT]

From [Unknown Contact. See original ticket]

I wrote

I hope not. chomp() should match the string in $?, no more or less.
  $/

Damn shift key.

Mike Guy

[p5pRT]

From @TimToady

Tom Christiansen writes​:
: >When I said, "I talked to" I meant that. I don't need the pointer - I know
: >how to handle it. But I wind up answering questions from people who do
: >not. In my experience most of them use chomp() so the confusion is
: >preventable. (After all you expect chomp() to get rid of line endings,
: >right?)
:
: I expect chomp() to remove one and only one terminating instance of the
: precise string to which $/ has been set; no more, no less. What were
: you expecting?

I expect people to expect Perl to do the right thing.

Larry

[p5pRT]

From @TimToady

M.J.T. Guy writes​:
: I think the "right" way of doing this is by providing some sort of
: filter apparatus on files. Things of this sort were discussed in
: the context of unicode. I don't recall where (if anywhere) that
: ended.

Ended? It hasn't started yet...

(Can you tell I've spent too much time rewriting the Camel book today? :-)

Yes, input filters should handle this. And a good case can be made
that the *default* input filter should handle it, along with UTF-8
recognition. It also has to be blazing fast, of course, along with
reading your mind. But that's Perl for the coarse. Or something like
that.

Larry

[p5pRT]

From [Unknown Contact. See original ticket]

​: I expect chomp() to remove one and only one terminating instance of the
​: precise string to which $/ has been set; no more, no less. What were
​: you expecting?

I expect people to expect Perl to do the right thing.

And that would be what, sniff around the stdio buffer the first time you
play with it and figure out what it smells like?

--tom

[p5pRT]

From @TimToady

Tom Christiansen writes​:
: >​: I expect chomp() to remove one and only one terminating instance of the
: >​: precise string to which $/ has been set; no more, no less. What were
: >​: you expecting?
:
: >I expect people to expect Perl to do the right thing.
:
: And that would be what, sniff around the stdio buffer the first time you
: play with it and figure out what it smells like?

Why do you say "you"? Did I say I expect Perl to do the right thing? :-)

Seriously, we are entering an era when dwimmerly action on input will
be a necessary evil. I could wish it were otherwise, but my supply of
divine fiats is low. And I don't think anyone else has enough fiats to
pull it off either. For the near future I only see a chaotic dance
around the UTF-8 strange attractor, in part because a lot of butterflies
are flapping their wings near the UTF-16 attractor instead. We're going
to live in interesting times, whether or not that's an ancient Chinese
curse.

Larry

[p5pRT]

From [Unknown Contact. See original ticket]

Tom Christiansen <tchrist@​jhereg.perl.com> writes​:

​: I expect chomp() to remove one and only one terminating instance of the
​: precise string to which $/ has been set; no more, no less. What were
​: you expecting?

I expect people to expect Perl to do the right thing.

And that would be what, sniff around the stdio buffer the first time you
play with it and figure out what it smells like?

That is far from daft. sv_gets() (the internals of readline) would know
what it had used to find the end of the line. It could leave the
information around for chomp to use.

But the "right thing" is
just to return \n as a logical newline however it was represented in the
buffer (unless in binmode of course). Then chomp'ing \n is fine.

--tom
--
Nick Ing-Simmons

[p5pRT]

From [Unknown Contact. See original ticket]

Nick Ing-Simmons <nick@​ing-simmons.net> writes​:
|| Tom Christiansen <tchrist@​jhereg.perl.com> writes​:
|| >>​: I expect chomp() to remove one and only one terminating instance of the
|| >>​: precise string to which $/ has been set; no more, no less. What were
|| >>​: you expecting?
|| >
|| >>I expect people to expect Perl to do the right thing.
|| >
|| >And that would be what, sniff around the stdio buffer the first time you
|| >play with it and figure out what it smells like?
||
|| That is far from daft. sv_gets() (the internals of readline) would know
|| what it had used to find the end of the line. It could leave the
|| information around for chomp to use.

But as soon as you have a program that opens multiple files of
differing formats, this breaks down. You end up with taint-like
tracing of strings to track which form of file each string came
from.

Which means that...

|| But the "right thing" is
|| just to return \n as a logical newline however it was represented in the
|| buffer (unless in binmode of course). Then chomp'ing \n is fine.

is really a much better choice. That just leaves the issue of
determining the right filtering to do for an output file so that
it matches the input file it is derived from or the target it is
being written to or whatever is the most significant issue -
which the programmer will have to deal with.

--
John Macdonald jmm@​jmm.pickering.elegant.com