What do we mean by "paragraph mode"? #16787

p5pRT · 2018-12-12T22:41:10Z

Migrated from rt.perl.org#133722 (status was 'resolved')

Searchable as RT133722$

p5pRT · 2018-12-12T22:41:11Z

From @jkeenan

Summary: This ticket replaces RT 133703 and proposes improved
documentation and testing for the so-called "paragraph mode", i.e.,
processing of records in a file while $/ set to the empty string.

I. Background:

In https://rt-archive.perl.org/perl5/Ticket/Display.html?id=133703, I submitted a
patch to sv.c intended to address a "Comparison result is always the
same" warning reported by LTGM.com analysis of the Perl 5 core
distribution. Dave Mitchell pointed out where the patch was wrong and,
in the course of his discussion, mentioned that the section of sv.c at
issue governed so-called "paragraph mode", that is, what happens when
you read a file when the input record separator has been set to an empty
string:

#####
$/ = '';
#####

While poking around in the code and the test suite, I became convinced:

(a) that I didn't understand "paragraph mode" very well;

(b) that "paragraph mode" wasn't well documented;

(c) that "paragraph mode" wasn't thoroughly tested in the core
distribution's test suite; and

(d) that as a consequence of (b) and (c), it might contain bugs.

I spent several days working on this. I no longer think "paragraph
mode" has bugs, but I'm more convinced that it is under-documented and
under-tested. In this RT I propose better documentation of paragraph
mode and additional tests.

II. Paragraph Mode as Currently Found in the Core Distribution

From this point forward I'll treat "paragraph mode" and "setting $/ to
an empty string" as equivalent.

In the core distribution $/ (or $INPUT_RECORD_SEPARATOR) is defined in
pod/perlvar.pod and is mainly discussed in perlfaq5 and perlfaq6.
Please see this attachment:

#####
paragraph_mode_in_core_distribution.pod
#####

... for a thorough discussion of paragraph mode in the core
distribution, including instances in code found in cpan/, ext/, dist/
and lib/. You can also find this discussion on the web at:

#####
http://thenceforward.net/perl/misc/paragraph_mode_in_core_distribution.html
#####

III. Proposed Additional Documentation

I believe that the best place to put additional discussion of paragraph
mode will be in perlfaq5. Please review the patch attached

#####
0001-More-detailed-explanation-of-paragraph-mode.patch
#####

Note that perlfaq5 is maintained upstream on CPAN, so once P5P is
on-board with the patch, I will submit it as a github issue at
https://github.com/perl-doc-cats/perlfaq/issues.

IV. Proposed Additional Testing

Please see the program attached:

#####
paragraph_mode.t
#####

To facilitate discussion, I've written this test program in a modern
style using Test::More, File::Temp, CPAN module Data::Dump and
subroutines. When we bring this into the core distribution, I'll adapt
it for inclusion under, say, t/op/, where I'll have to use only the
testing functions provided by t/test.pl.

What is important for discussion now is: Do these tests thoroughly
cover what we mean by "paragraph mode"?

Thank you very much.
Jim Keenan

p5pRT · 2018-12-12T22:41:11Z

From @jkeenan

paragraph_mode_in_core_distribution.pod

p5pRT · 2018-12-12T22:41:11Z

From @jkeenan

0001-More-detailed-explanation-of-paragraph-mode.patch

From dc1d6b22a64e9ddbf003204beef033bf320cbe81 Mon Sep 17 00:00:00 2001
From: James E Keenan <jkeenan@cpan.org>
Date: Wed, 12 Dec 2018 16:52:00 -0500
Subject: [PATCH] More detailed explanation of "paragraph mode"

---
 lib/perlfaq5.pod | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/lib/perlfaq5.pod b/lib/perlfaq5.pod
index 7464b26..96470e7 100644
--- a/lib/perlfaq5.pod
+++ b/lib/perlfaq5.pod
@@ -1179,6 +1179,16 @@ C<"\n\n"> to accept empty paragraphs.
 Note that a blank line must have no blanks in it. Thus
 S<C<"fred\n \nstuff\n\n">> is one paragraph, but C<"fred\n\nstuff\n\n"> is two.
 
+When C<$/> is set to C<""> -- so-called I<paragraph mode> -- and the entire
+file is read in with that setting, any sequence of consecutive newlines
+C<"\n\n"> at the beginning of the file is discarded.  With the exception of
+the final record in the file, each sequence of characters ending in two or
+more newlines is treated as one record and is read in to end in exactly two
+newlines.  If the last record in the file ends in zero or one consecutive
+newlines, that record is read in with that number of newlines.  If the last
+record ends in two or more consecutive newlines, it is read in with two
+newlines like all preceding records.
+
 =head2 How can I read a single character from a file? From the keyboard?
 X<getc> X<file, reading one character at a time>
 
-- 
2.17.1

p5pRT · 2018-12-12T22:41:11Z

From @jkeenan

use strict;
use warnings;
use Test::More;
use File::Temp qw( tempfile );
use Data::Dumper;
use Data::Dump qw( dd pp );

# Test paragraph mode in two ways:
# first, in the style of t/base/rs.t, by reading one record at a time
# and comparing it to an expected value,
# then, after seek-ing back to start of file, by reading all the records
# into an array and then comparing the array to an expected value.

my ($OUT, $filename, @chunks, @expected, $msg);
my $testcount = 0;

{
note("'Well behaved' files");
# We start with files whose "paragraphs" contain no internal newlines.
@chunks = (
join('' => ( 1..3 )),
join('' => ( 4..6 )),
join('' => ( 7..9 )),
10
);

{
$msg = "Case " . ++$testcount . ": 'Well behaved' file: >= 2 newlines between text blocks\n";
$msg .= " no internal newlines; no final newline";
note($msg);

($OUT, $filename) = open_tempfile();
print $OUT "$_\n" for (
$chunks[0],
("") x 1,
$chunks[1],
("") x 2,
$chunks[2],
("") x 3,
);
print $OUT $chunks[3];
close $OUT or die;

@expected = (
"$chunks[0]\n\n",
"$chunks[1]\n\n",
"$chunks[2]\n\n",
$chunks[3],
);
local $/ = '';
perform_tests($filename, \@expected);
}

{
$msg = "Case " . ++$testcount . ": 'Well behaved' file: >= 2 newlines between text blocks\n";
$msg .= " no internal newlines; 1 final newline";
note($msg);

($OUT, $filename) = open_tempfile();
print $OUT "$_\n" for (
$chunks[0],
("") x 1,
$chunks[1],
("") x 2,
$chunks[2],
("") x 3,
$chunks[3],
);
close $OUT or die;

@expected = (
"$chunks[0]\n\n",
"$chunks[1]\n\n",
"$chunks[2]\n\n",
"$chunks[3]\n",
);
local $/ = '';
perform_tests($filename, \@expected);
}

{
$msg = "Case " . ++$testcount . ": 'Well behaved' file: >= 2 newlines between text blocks\n";
$msg .= " no internal newlines; 2 final newlines";
note($msg);

($OUT, $filename) = open_tempfile();
print $OUT "$_\n" for (
$chunks[0],
("") x 1,
$chunks[1],
("") x 2,
$chunks[2],
("") x 3,
$chunks[3],
("") x 1,
);
close $OUT or die;

@expected = (
"$chunks[0]\n\n",
"$chunks[1]\n\n",
"$chunks[2]\n\n",
"$chunks[3]\n\n",
);
local $/ = '';
perform_tests($filename, \@expected);
}

{
$msg = "Case " . ++$testcount . ": 'Well behaved' file: >= 2 newlines between text blocks\n";
$msg .= " no internal newlines; 3 final newlines";
note($msg);

($OUT, $filename) = open_tempfile();
print $OUT "$_\n" for (
$chunks[0],
("") x 1,
$chunks[1],
("") x 2,
$chunks[2],
("") x 3,
$chunks[3],
("") x 2,
);
close $OUT or die;

@expected = (
"$chunks[0]\n\n",
"$chunks[1]\n\n",
"$chunks[2]\n\n",
"$chunks[3]\n\n",
);
local $/ = '';
perform_tests($filename, \@expected);
}
}

{
note("'Misbehaving' files");
# We continue with files whose "paragraphs" contain internal newlines.
@chunks = (
join('' => ( 1, 2, "\n", 3 )),
join('' => ( 4, 5, " \n", 6 )),
join('' => ( 7, 8, " \t\n", 9 )),
10
);

{
$msg = "Case " . ++$testcount . ": 'Misbehaving' file: >= 2 newlines between text blocks\n";
$msg .= " no internal newlines; no final newline";
note($msg);

($OUT, $filename) = open_tempfile();
print $OUT "$_\n" for (
$chunks[0],
("") x 1,
$chunks[1],
("") x 2,
$chunks[2],
("") x 3,
);
print $OUT $chunks[3];
close $OUT or die;

@expected = (
"$chunks[0]\n\n",
"$chunks[1]\n\n",
"$chunks[2]\n\n",
$chunks[3],
);
local $/ = '';
perform_tests($filename, \@expected);
}

{
$msg = "Case " . ++$testcount . ": 'Misbehaving' file: >= 2 newlines between text blocks\n";
$msg .= " no internal newlines; 1 final newline";
note($msg);

($OUT, $filename) = open_tempfile();
print $OUT "$_\n" for (
$chunks[0],
("") x 1,
$chunks[1],
("") x 2,
$chunks[2],
("") x 3,
$chunks[3],
);
close $OUT or die;

@expected = (
"$chunks[0]\n\n",
"$chunks[1]\n\n",
"$chunks[2]\n\n",
"$chunks[3]\n",
);
local $/ = '';
perform_tests($filename, \@expected);
}

{
$msg = "Case " . ++$testcount . ": 'Misbehaving' file: >= 2 newlines between text blocks\n";
$msg .= " no internal newlines; 2 final newlines";
note($msg);

($OUT, $filename) = open_tempfile();
print $OUT "$_\n" for (
$chunks[0],
("") x 1,
$chunks[1],
("") x 2,
$chunks[2],
("") x 3,
$chunks[3],
("") x 1,
);
close $OUT or die;

@expected = (
"$chunks[0]\n\n",
"$chunks[1]\n\n",
"$chunks[2]\n\n",
"$chunks[3]\n\n",
);
local $/ = '';
perform_tests($filename, \@expected);
}

{
$msg = "Case " . ++$testcount . ": 'Misbehaving' file: >= 2 newlines between text blocks\n";
$msg .= " no internal newlines; 3 final newlines";
note($msg);

($OUT, $filename) = open_tempfile();
print $OUT "$_\n" for (
$chunks[0],
("") x 1,
$chunks[1],
("") x 2,
$chunks[2],
("") x 3,
$chunks[3],
("") x 2,
);
close $OUT or die;

@expected = (
"$chunks[0]\n\n",
"$chunks[1]\n\n",
"$chunks[2]\n\n",
"$chunks[3]\n\n",
);
local $/ = '';
perform_tests($filename, \@expected);
}
}

{
note("'Badly behaved' files starting with newlines");
# We continue with files which start with newlines
# but whose "paragraphs" contain no internal newlines.
# We'll set our expectation that the leading newlines will get trimmed off
# and everything else will proceed normally.

@chunks = (
join('' => ( 1..3 )),
join('' => ( 4..6 )),
join('' => ( 7..9 )),
10
);

{
$msg = "Case " . ++$testcount . ": 'Badly behaved' file: leading newlines";
note($msg);

($OUT, $filename) = open_tempfile();
print $OUT "\n\n\n";
print $OUT "$_\n" for (
$chunks[0],
("") x 1,
$chunks[1],
("") x 2,
$chunks[2],
("") x 3,
);
print $OUT $chunks[3];
close $OUT or die;

@expected = (
"$chunks[0]\n\n",
"$chunks[1]\n\n",
"$chunks[2]\n\n",
$chunks[3],
);
local $/ = '';
perform_tests($filename, \@expected);
}

{
$msg = "Case " . ++$testcount . ": 'Badly behaved' file: leading newlines";
note($msg);

($OUT, $filename) = open_tempfile();
print $OUT "\n\n\n";
print $OUT "$_\n" for (
$chunks[0],
("") x 1,
$chunks[1],
("") x 2,
$chunks[2],
("") x 3,
$chunks[3],
);
close $OUT or die;

@expected = (
"$chunks[0]\n\n",
"$chunks[1]\n\n",
"$chunks[2]\n\n",
"$chunks[3]\n",
);
local $/ = '';
perform_tests($filename, \@expected);
}

{
$msg = "Case " . ++$testcount . ": 'Badly behaved' file: leading newlines";
note($msg);

($OUT, $filename) = open_tempfile();
print $OUT "\n\n\n";
print $OUT "$_\n" for (
$chunks[0],
("") x 1,
$chunks[1],
("") x 2,
$chunks[2],
("") x 3,
$chunks[3],
("") x 1,
);
close $OUT or die;

@expected = (
"$chunks[0]\n\n",
"$chunks[1]\n\n",
"$chunks[2]\n\n",
"$chunks[3]\n\n",
);
local $/ = '';
perform_tests($filename, \@expected);
}

{
$msg = "Case " . ++$testcount . ": 'Badly behaved' file: leading newlines";
note($msg);

($OUT, $filename) = open_tempfile();
print $OUT "\n\n\n";
print $OUT "$_\n" for (
$chunks[0],
("") x 1,
$chunks[1],
("") x 2,
$chunks[2],
("") x 3,
$chunks[3],
("") x 2,
);
close $OUT or die;

@expected = (
"$chunks[0]\n\n",
"$chunks[1]\n\n",
"$chunks[2]\n\n",
"$chunks[3]\n\n",
);
local $/ = '';
perform_tests($filename, \@expected);
}
}

{
note("'Very badly behaved' files starting with newlines");
# We continue with files which start with newlines
# and whose "paragraphs" contain internal newlines.
# We'll set our expectation that the leading newlines will get trimmed off
# and everything else will proceed normally.

@chunks = (
join('' => ( 1, 2, "\n", 3 )),
join('' => ( 4, 5, " \n", 6 )),
join('' => ( 7, 8, " \t\n", 9 )),
10
);

{
$msg = "Case " . ++$testcount . ": 'Very badly behaved' file: leading newlines";
$msg .= " internal newlines";
note($msg);