Using .lines on generated data is wrong (traps)

[AlexDaniel]

Let's make a file:

perl6 -e 'spurt ‘file’, “foox\nfooy\rbar\nfooz\n”'

It is fair to say (on linux at least) that this file has three lines:

foox
fooy\rbar
fooz

And any proper command line tool in existence will treat it as such. wc -l says 3, grep foo will give the whole line also (your terminal may print it in a weird way, but that's still treated as a whole line).

So let's try reading this file in perl 6:

perl6 -e 'dd $_ for lines' < file

"foox"
"fooy\rbar"
"fooz"

↑ This is entirely right.

But here's a trap:

perl6 -e 'dd $_ for slurp.lines' < file

"foox"
"fooy"
"bar"
"fooz"

It turns out that lines and slurp.lines work differently, which is really surprising. Also, I managed to get into this trap at least once.

[AlexDaniel]

Also note that Proc::Async.stdout.lines is totally useless in this regard.

my $proc = Proc::Async.new: ‘cat’, ‘file’; # same file as in OP
react {
    whenever $proc.stdout.lines { dd $_ }
    whenever $proc.start { done }
}

"foox"
"fooy"
"bar"
"fooz"

So if you're using Proc::Async, you're doomed. There's RT #131923 which asks for customizable nl-in or split, but we don't have that yet.

[AlexDaniel]

From our docs:

If :translate-nl is set to True (default value), OS-specific newline terminators (e.g. \r\n on Windows) will be automatically translated to \n.

Let's see if it helps.

my $proc = Proc::Async.new: ‘cat’, ‘file’, :!translate-nl; # same file as in OP
react {
    whenever $proc.stdout.lines { dd $_ }
    whenever $proc.start { done }
}

"foox"
"fooy"
"bar"
"fooz"

Nope.

[zoffixznet]

Is this issue mistitled?

I don't quite understand where the wrongness of calling .lines lies.

[AlexDaniel]

Mistitled or not, I think this is now resolved.

[zoffixznet]

👍