Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Filter::Simple is confused by two divisions #20489

Open
wlmb opened this issue Nov 6, 2022 · 6 comments
Open

Filter::Simple is confused by two divisions #20489

wlmb opened this issue Nov 6, 2022 · 6 comments

Comments

@wlmb
Copy link

wlmb commented Nov 6, 2022

To: perl5-porters@perl.org
Subject: Filter::Simple is confused by two divisions
Message-Id: 5.36.0_128371_1667697003@yapaque.fis.unam.mx
Reply-To: mochan@fis.unam.mx
Cc: mochan@yapaque.fis.unam.mx
From: mochan@fis.unam.mx

This is a bug report for perl from mochan@fis.unam.mx,
generated with the help of perlbug 1.42 running under perl 5.36.0.

[Please describe your issue here]

Someone found the following bug while running PDL::NiceSlice. Here is
a simplified version.

Consider a trivial sourcefilter MyFilter.pm that does nothing but
report its input after removing comments and after putting them back

package MyFilter;
use v5.36;
use Filter::Simple;
FILTER_ONLY
code_no_comments => sub { say "Code_no_comments:\n$_" },
all => sub { say "All:\n$_" };
1;

Now I use the filter in the following program:

use v5.36;
use lib ".";
use MyFilter;
my $x=1;
say $x/2/3;

When I run it I get the following output

$ perl rem1.pl
Code:
my $x=1;
say $x3;

All:
my $x=1;
say $x/2/3;

0.166666666666667

The code fragment '/2/' is erroneously removed from the code before being fed to
the filter. However, if I change the name of the variable to something
else, as $y, there is no error. But if I change it to something ending
in x, as in $yx, the error reappears. Whatever is between the two
slashes is irrelevant. The second could be a in a comment in another
line and the error remains, as in

use v5.36;
use lib ".";
use MyFilter;
my $x=1;
say $x/2

#/3;

Output:

~$ perl rem1.pl
Code:
my $x=1;
say $x3;

All:
my $x=1;
say $x/2

#/3;

0.5

Best regards,

Luis

[Please do not change anything below this line]

Flags:
category=library
severity=medium
module=Filter::Simple

Site configuration information for perl 5.36.0:

Configured by mochan at Mon Jun 13 13:52:52 CDT 2022.

Summary of my perl5 (revision 5 version 36 subversion 0) configuration:

Platform:
osname=linux
osvers=5.17.0-1-amd64
archname=x86_64-linux
uname='linux yapaque 5.17.0-1-amd64 #1 smp preempt debian 5.17.3-1 (2022-04-18) x86_64 gnulinux '
config_args='-de -Dprefix=/home/mochan/perl5/perlbrew/perls/perl-5.36.0 -Aeval:scriptdir=/home/mochan/perl5/perlbrew/perls/perl-5.36.0/bin'
hint=recommended
useposix=true
d_sigaction=define
useithreads=undef
usemultiplicity=undef
use64bitint=define
use64bitall=define
uselongdouble=undef
usemymalloc=n
default_inc_excludes_dot=define
Compiler:
cc='cc'
ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2'
optimize='-O2'
cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
ccversion=''
gccversion='11.3.0'
gccosandvers=''
intsize=4
longsize=8
ptrsize=8
doublesize=8
byteorder=12345678
doublekind=3
d_longlong=define
longlongsize=8
d_longdbl=define
longdblsize=16
longdblkind=3
ivtype='long'
ivsize=8
nvtype='double'
nvsize=8
Off_t='off_t'
lseeksize=8
alignbytes=8
prototype=define
Linker and Libraries:
ld='cc'
ldflags =' -fstack-protector-strong -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib
libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
libc=libc-2.33.so
so=so
useshrplib=false
libperl=libperl.a
gnulibc_version='2.33'
Dynamic Linking:
dlsrc=dl_dlopen.xs
dlext=so
d_dlsymun=undef
ccdlflags='-Wl,-E'
cccdlflags='-fPIC'
lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'

@inc for perl 5.36.0:
/home/mochan/perl5/perlbrew/perls/perl-5.36.0/lib/site_perl/5.36.0/x86_64-linux
/home/mochan/perl5/perlbrew/perls/perl-5.36.0/lib/site_perl/5.36.0
/home/mochan/perl5/perlbrew/perls/perl-5.36.0/lib/5.36.0/x86_64-linux
/home/mochan/perl5/perlbrew/perls/perl-5.36.0/lib/5.36.0

Environment for perl 5.36.0:
HOME=/home/mochan
LANG=en_US.UTF-8
LANGUAGE=en_US:en
LD_LIBRARY_PATH (unset)
LOGDIR (unset)
PATH=/home/mochan/bin:/home/mochan/.cargo/bin:/home/mochan/gems/bin:/home/mochan/bin:/home/mochan/perl5/perlbrew/bin:/home/mochan/perl5/perlbrew/perls/perl-5.36.0/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin:/home/mochan/rakudo/rakudo-star-2020.01/install/bin:/home/mochan/rakudo/rakudo-star-2020.01/install/share/perl6/site/bin
PERLBREW_HOME=/home/mochan/.perlbrew
PERLBREW_MANPATH=/home/mochan/perl5/perlbrew/perls/perl-5.36.0/man
PERLBREW_PATH=/home/mochan/perl5/perlbrew/bin:/home/mochan/perl5/perlbrew/perls/perl-5.36.0/bin
PERLBREW_PERL=perl-5.36.0
PERLBREW_ROOT=/home/mochan/perl5/perlbrew
PERLBREW_SHELLRC_VERSION=0.86
PERLBREW_VERSION=0.86
PERL_BADLANG (unset)
PERL_CPANM_OPT=-M https://cpan.metacpan.org/
SHELL=/bin/bash
@jkeenan
Copy link
Contributor

jkeenan commented Nov 8, 2022

Here is what I get when I append more division operations to the statement.

3 divisions

[gh-20489-filter-simple] 546 $ cat three-divisions.pl 
use v5.36;
use lib ".";
use MyFilter;
my $x=144;
say $x/2/3/4;
[gh-20489-filter-simple] 547 $ perl three-divisions.pl 
Code_no_comments:
my $x=144;
say $x3/4;

All:
my $x=144;
say $x/2/3/4;

6

4 divisions

[gh-20489-filter-simple] 548 $ cat four-divisions.pl 
use v5.36;
use lib ".";
use MyFilter;
my $x=1440;
say $x/2/3/4/5;
[gh-20489-filter-simple] 549 $ perl four-divisions.pl 
Code_no_comments:
my $x=1440;
say $x35;

All:
my $x=1440;
say $x/2/3/4/5;

12

@jkeenan
Copy link
Contributor

jkeenan commented Nov 8, 2022

Someone found the following bug while running PDL::NiceSlice. Here is a simplified version.

Consider a trivial sourcefilter MyFilter.pm that does nothing but report its input after removing comments and after putting them back

package MyFilter;
use v5.36;
use Filter::Simple;
FILTER_ONLY
code_no_comments => sub { say "Code_no_comments:\n$_" },
all => sub { say "All:\n$_" };
1;

Now I use the filter in the following program:

use v5.36;
use lib ".";
use MyFilter;
my $x=1;
say $x/2/3;

When I run it I get the following output

$ perl rem1.pl
Code:
my $x=1;
say $x3;

All:
my $x=1;
say $x/2/3;

0.166666666666667

The problem seems to be restricted to the / operator. I tried your module with 3 other programs using +, - and * and did not get a problem.

[gh-20489-filter-simple] 551 $ cat two-additions.pl 
use v5.36;
use lib ".";
use MyFilter;
my $x=1;
say $x+2+3;
[gh-20489-filter-simple] 552 $ perl two-additions.pl 
Code_no_comments:
my $x=1;
say $x+2+3;

All:
my $x=1;
say $x+2+3;

6
[gh-20489-filter-simple] 553 $ cat two-subtractions.pl 
use v5.36;
use lib ".";
use MyFilter;
my $x=1;
say $x-2-3;
[gh-20489-filter-simple] 554 $ perl two-subtractions.pl 
Code_no_comments:
my $x=1;
say $x-2-3;

All:
my $x=1;
say $x-2-3;

-4
[gh-20489-filter-simple] 555 $ cat two-multiplications.pl 
use v5.36;
use lib ".";
use MyFilter;
my $x=1;
say $x*2*3;
[gh-20489-filter-simple] 556 $ perl two-multiplications.pl 
Code_no_comments:
my $x=1;
say $x*2*3;

All:
my $x=1;
say $x*2*3;

6

@jkeenan
Copy link
Contributor

jkeenan commented Nov 8, 2022

Someone found the following bug while running PDL::NiceSlice. Here is a simplified version.

Do you have a URL for that bug report?

@wlmb
Copy link
Author

wlmb commented Nov 8, 2022

Thanks for confirming and exploring further cases. Regards.

@wlmb
Copy link
Author

wlmb commented Nov 8, 2022 via email

@tonycoz
Copy link
Contributor

tonycoz commented Nov 9, 2022

For me, the $x could be any variable name, it wasn't limited to $x:

$ ~/perl/v5.36.0-dbg-san/bin/perl 20489.pl 
Code_no_comments:
my $z=1;
say $z3;

All:
my $z=1;
say $z/2/3;

0.166666666666667

From what I can tell, Filter::Simple is treating the /3/ as a regular expression. If you dump the string as hex:

$ ~/perl/v5.36.0-dbg-san/bin/perl 20489.pl 
Code_no_comments:
my $z=1;
say $z3;

6d7920247a3d313b0a73617920247a1c000000001c333b0a
All:
my $z=1;
say $z/2/3;

0.166666666666667

You can confirm that by adding as regex callback:

$ ~/perl/v5.36.0-dbg-san/bin/perl 20489.pl 
Code_no_comments:
my $z=1;
say $z3;

Regex:
2
All:
my $z=1;
say $z/2/3;

0.166666666666667

The /3/ has been replaced by "\x1C\0\0\0\1\x1c" which is the documented behaviour for quotelikes in the Filter::Simple documentation:

    That is, a 'code...' filter *replaces* each quoted string, quotelike,
    regex, POD, and __DATA__ section with a placeholder. The delimiters of
    this placeholder are the contents of the $; variable at the time the
    filter is applied (normally "\034"). The remaining four bytes are a
    unique identifier for the component being replaced.

This doesn't appear to be a new issue, I see it as far back as 5.10 (after lowering the use vXX;):

$ ~/perl/5.10.0-debug/bin/perl 20489.pl
Code_no_comments:
my $x=1;
say $x3;

Regex:
2
All:
my $x=1;
say $x/2/3;

0.166666666666667

I see the same output with 5.34, which the author in the original thread upgraded from.

The only way to fix that I can see is to make Filter::Simple a full parser rather than the "guess and hope" current implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jkeenan @tonycoz @wlmb