Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Perl build process leaves random files in system #16412

Closed
p5pRT opened this issue Feb 10, 2018 · 21 comments
Closed

Perl build process leaves random files in system #16412

p5pRT opened this issue Feb 10, 2018 · 21 comments

Comments

@p5pRT
Copy link

@p5pRT p5pRT commented Feb 10, 2018

Migrated from rt.perl.org#132849 (status was 'resolved')

Searchable as RT132849$

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 10, 2018

From @dur-randir

Created by @dur-randir

Starting with the commit c0e3b4b, perl build process now leaves (on
some systems) files outside of it's build tree​:

dorothy at ~/afl-bisect ±(c0e3b4b) ❯ ls -al /home/core
total 32
drwxrwxrwx 2 root root 24576 Aug 9 2017 .
drwxr-xr-x 5 root root 4096 May 19 2017 ..

dorothy at ~/afl-bisect ±(c0e3b4b) ❯ git clean -dfx && ./Configure
-de -Dusedevel && make -j20 test_prep
<snip>
cd t && (rm -f perl; /bin/ln -s ../perl perl)

drwxrwxrwx 2 root root 24576 Feb 11 01​:22 .
drwxr-xr-x 5 root root 4096 May 19 2017 ..
-rw------- 1 afl afl 19984384 Feb 11 01​:22 core.40209
-rw------- 1 afl afl 16064512 Feb 11 01​:22 core.40211
-rw------- 1 afl afl 15572992 Feb 11 01​:22 core.40223
-rw------- 1 afl afl 15327232 Feb 11 01​:22 core.40233
-rw------- 1 afl afl 15294464 Feb 11 01​:22 core.40247
-rw------- 1 afl afl 15294464 Feb 11 01​:22 core.40267
-rw------- 1 afl afl 15708160 Feb 11 01​:22 core.40352
-rw------- 1 afl afl 15101952 Feb 11 01​:22 core.40392
-rw------- 1 afl afl 15101952 Feb 11 01​:22 core.40479

This pollutes users' systems. This leads to things like
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=132843. This can break in
other ways on restricted systems.

I do not find this behavior nor desirable nor sane.

Perl Info

Flags:
    category=library
    severity=high
    module=Storable

Site configuration information for perl 5.27.9:

Configured by root at Sun Feb 11 01:07:53 MSK 2018.

Summary of my perl5 (revision 5 version 27 subversion 9) configuration:
  Commit id: c0e3b4b51cabf15ed8fc5f564dfeea31c25f5239
  Platform:
    osname=linux
    osvers=4.9.0-5-amd64
    archname=x86_64-linux
    uname='linux dorothy 4.9.0-5-amd64 #1 smp debian 4.9.65-3+deb9u2
(2018-01-04) x86_64 gnulinux '
    config_args='-de -Dusedevel'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=undef
    usemultiplicity=undef
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='cc'
    ccflags ='-fwrapv -fno-strict-aliasing -pipe
-fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2'
    optimize='-O2'
    cppflags='-fwrapv -fno-strict-aliasing -pipe
-fstack-protector-strong -I/usr/local/include'
    ccversion=''
    gccversion='6.3.0 20170516'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags =' -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib
/usr/lib/gcc/x86_64-linux-gnu/6/include-fixed
/usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu
/lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
    libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.24.so
    so=so
    useshrplib=false
    libperl=libperl.a
    gnulibc_version='2.24'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E'
    cccdlflags='-fPIC'
    lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'



@INC for perl 5.27.9:
    lib
    /usr/local/lib/perl5/site_perl/5.27.9/x86_64-linux
    /usr/local/lib/perl5/site_perl/5.27.9
    /usr/local/lib/perl5/5.27.9/x86_64-linux
    /usr/local/lib/perl5/5.27.9


Environment for perl 5.27.9:
    HOME=/home/afl
    LANG=en_US.UTF-8
    LANGUAGE=en_US:en
    LC_CTYPE=en_US.UTF-8
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/afl/perlbrew/bin:/home/afl/perlbrew/perls/perl-5.20.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
    PERLBREW_BASHRC_VERSION=0.78
    PERLBREW_HOME=/home/afl/.perlbrew
    PERLBREW_MANPATH=/home/afl/perlbrew/perls/perl-5.20.2/man
    PERLBREW_PATH=/home/afl/perlbrew/bin:/home/afl/perlbrew/perls/perl-5.20.2/bin
    PERLBREW_PERL=perl-5.20.2
    PERLBREW_ROOT=/home/afl/perlbrew
    PERLBREW_VERSION=0.78
    PERL_BADLANG (unset)
    SHELL=/usr/bin/zsh

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 11, 2018

From @jkeenan

On Sat, 10 Feb 2018 22​:29​:20 GMT, randir wrote​:

This is a bug report for perl from sergey.aleynikov@​gmail.com,
generated with the help of perlbug 1.41 running under perl 5.27.9.

-----------------------------------------------------------------
[Please describe your issue here]

Starting with the commit c0e3b4b, perl build process now leaves (on
some systems) files outside of it's build tree​:

dorothy at ~/afl-bisect ±(c0e3b4b) ❯ ls -al /home/core
total 32
drwxrwxrwx 2 root root 24576 Aug 9 2017 .
drwxr-xr-x 5 root root 4096 May 19 2017 ..

dorothy at ~/afl-bisect ±(c0e3b4b) ❯ git clean -dfx && ./Configure
-de -Dusedevel && make -j20 test_prep
<snip>
cd t && (rm -f perl; /bin/ln -s ../perl perl)

drwxrwxrwx 2 root root 24576 Feb 11 01​:22 .
drwxr-xr-x 5 root root 4096 May 19 2017 ..
-rw------- 1 afl afl 19984384 Feb 11 01​:22 core.40209
-rw------- 1 afl afl 16064512 Feb 11 01​:22 core.40211
-rw------- 1 afl afl 15572992 Feb 11 01​:22 core.40223
-rw------- 1 afl afl 15327232 Feb 11 01​:22 core.40233
-rw------- 1 afl afl 15294464 Feb 11 01​:22 core.40247
-rw------- 1 afl afl 15294464 Feb 11 01​:22 core.40267
-rw------- 1 afl afl 15708160 Feb 11 01​:22 core.40352
-rw------- 1 afl afl 15101952 Feb 11 01​:22 core.40392
-rw------- 1 afl afl 15101952 Feb 11 01​:22 core.40479

This pollutes users' systems. This leads to things like
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=132843. This can break in
other ways on restricted systems.

I do not find this behavior nor desirable nor sane.

While I share your concern about failure to clean up files intended to be temporary, I'm having trouble replicating your results.

At commit 199fc8c I cleaned my work directory, then called Configure and make test_prep as I ordinarily do. When those completed, I checked the following directories for un-cleaned-up files​:

/
/tmp/
/home/
~/
~/tmp/

I didn't find any core.* files in any of those locations. Can you tell me where I should have been looking? Or explain a bit more about the circumstances in which these files might not be cleaned up?

Thank you very much.
--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 11, 2018

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 11, 2018

From @dur-randir

On Sat, 10 Feb 2018 17​:07​:56 -0800, jkeenan wrote​:

While I share your concern about failure to clean up files intended to
be temporary, I'm having trouble replicating your results.

I didn't find any core.* files in any of those locations. Can you
tell me where I should have been looking? Or explain a bit more about
the circumstances in which these files might not be cleaned up?

So, here's a full picture.

On linux systems, location for core files is controlled by a system-wide tunable in /proc/sys/kernel/core_pattern. Default value for it varies across different Linux distros. But, since it it's system-wide, it can only be set by root and software can't rely on it's specific value.

Then, there's a second layer of defense - to allow programs some control, whether they want to leave core files after them, or not. There is a per-process limit, named 'core limit', which when set to 0 disables generating core files for this process altogether. It can be set, for example, in a shell by executing 'limit -c 0' to suppress core files from all subsequently run commands from this shell (or 'limit -c unlimited' to enable core generation for processes of all sizes).

So, to reproduce this, you should​:

- save somewhere value of /proc/sys/kernel/core_pattern (to restore it later)
- set, as root, that tunable to '/home/core/%p.core', where %p means process pid (or anywhere else you'd like it to point for a test)
- execute 'ulimit -c unlimited' in a shell where you'd run 'make'
- build perl

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 11, 2018

From @jkeenan

On Sun, 11 Feb 2018 10​:11​:34 GMT, randir wrote​:

On Sat, 10 Feb 2018 17​:07​:56 -0800, jkeenan wrote​:

While I share your concern about failure to clean up files intended
to
be temporary, I'm having trouble replicating your results.

I didn't find any core.* files in any of those locations. Can you
tell me where I should have been looking? Or explain a bit more
about
the circumstances in which these files might not be cleaned up?

So, here's a full picture.

On linux systems, location for core files is controlled by a system-
wide tunable in /proc/sys/kernel/core_pattern. Default value for it
varies across different Linux distros. But, since it it's system-wide,
it can only be set by root and software can't rely on it's specific
value.

Then, there's a second layer of defense - to allow programs some
control, whether they want to leave core files after them, or not.
There is a per-process limit, named 'core limit', which when set to 0
disables generating core files for this process altogether. It can be
set, for example, in a shell by executing 'limit -c 0' to suppress
core files from all subsequently run commands from this shell (or
'limit -c unlimited' to enable core generation for processes of all
sizes).

So, to reproduce this, you should​:

- save somewhere value of /proc/sys/kernel/core_pattern (to restore it
later)
- set, as root, that tunable to '/home/core/%p.core', where %p means
process pid (or anywhere else you'd like it to point for a test)
- execute 'ulimit -c unlimited' in a shell where you'd run 'make'
- build perl

Thanks for that explanation. I was previously unaware of /proc/sys/kernel/core_pattern. On Ubuntu that pipes a crash report to the 'apport' program described here​: https://wiki.ubuntu.com/Apport.

Of course, the point you made in your original post is still a concern. I don't know how to address it best.

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 12, 2018

From @tonycoz

On Sun, 11 Feb 2018 02​:11​:34 -0800, randir wrote​:

On Sat, 10 Feb 2018 17​:07​:56 -0800, jkeenan wrote​:

While I share your concern about failure to clean up files intended
to
be temporary, I'm having trouble replicating your results.

I didn't find any core.* files in any of those locations. Can you
tell me where I should have been looking? Or explain a bit more
about
the circumstances in which these files might not be cleaned up?

So, here's a full picture.

On linux systems, location for core files is controlled by a system-
wide tunable in /proc/sys/kernel/core_pattern. Default value for it
varies across different Linux distros. But, since it it's system-wide,
it can only be set by root and software can't rely on it's specific
value.

Then, there's a second layer of defense - to allow programs some
control, whether they want to leave core files after them, or not.
There is a per-process limit, named 'core limit', which when set to 0
disables generating core files for this process altogether. It can be
set, for example, in a shell by executing 'limit -c 0' to suppress
core files from all subsequently run commands from this shell (or
'limit -c unlimited' to enable core generation for processes of all
sizes).

In 7ec112d I've added code to dist/Storable/stacksize to "ulimit -c 0"* before running the stack overflow tests.

Please let me know if this fixes the problem for you.

If it doesn't I'll disable this probing by default.

Thanks,
Tony

* where available

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 12, 2018

From @dur-randir

On Sun, 11 Feb 2018 16​:50​:16 -0800, tonyc wrote​:

Please let me know if this fixes the problem for you.

If it doesn't I'll disable this probing by default.

Now it leaves no traces on linux. But then I've checked windows x64 and x86 builds using VS 2017. x64 passed flawlessly, but for x86 i've got this​:

- http​://i.imgur.com/9WIXn60.png
- (after pressing close) http​://i.imgur.com/ehVen2N.png

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 12, 2018

From @dur-randir

On Mon, 12 Feb 2018 12​:27​:01 -0800, randir wrote​:

x64 passed flawlessly, but for x86 i've
got this​:

I've done this again from a clean checkout and now also compiles. Can it be that 'nmake clean' leaves something behind, as I've built 64-bit first and then 32-bit?

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 12, 2018

From @xenu

On Mon, 12 Feb 2018, at 22​:14, Sergey Aleynikov via RT wrote​:

On Mon, 12 Feb 2018 12​:27​:01 -0800, randir wrote​:

x64 passed flawlessly, but for x86 i've
got this​:

I've done this again from a clean checkout and now also compiles. Can it
be that 'nmake clean' leaves something behind, as I've built 64-bit
first and then 32-bit?

My experience says that (n)make clean can't be trusted. git clean -dxf is
the most foolproof way to clean source tree.

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 12, 2018

From @khwilliamson

On 02/12/2018 01​:27 PM, Sergey Aleynikov via RT wrote​:

On Sun, 11 Feb 2018 16​:50​:16 -0800, tonyc wrote​:

Please let me know if this fixes the problem for you.

If it doesn't I'll disable this probing by default.

Now it leaves no traces on linux. But then I've checked windows x64 and x86 builds using VS 2017. x64 passed flawlessly, but for x86 i've got this​:

- http​://i.imgur.com/9WIXn60.png
- (after pressing close) http​://i.imgur.com/ehVen2N.png

I was getting those errors last week on dromedary's win32 (whatever that
is, maybe a VM?), and I "solved" it by cloning a new repository. I'm
pretty sure it happened before Storable was pushed to blead. So some
file, maybe it's just config data, is getting left around by a
git clean -dfx
that doesn't show up under git status.

I looked up error 5 and it was something about no access rights IIRC. I
don't know about the high-order 0xC.

---
via perlbug​: queue​: perl5 status​: open
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=132849

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 12, 2018

From @tonycoz

On Mon, Feb 12, 2018 at 12​:27​:01PM -0800, Sergey Aleynikov via RT wrote​:

On Sun, 11 Feb 2018 16​:50​:16 -0800, tonyc wrote​:

Please let me know if this fixes the problem for you.

If it doesn't I'll disable this probing by default.

Now it leaves no traces on linux. But then I've checked windows x64 and x86 builds using VS 2017. x64 passed flawlessly, but for x86 i've got this​:

- http​://i.imgur.com/9WIXn60.png
- (after pressing close) http​://i.imgur.com/ehVen2N.png

I expected the code I'm using to suppress that error worked on both 32
and 64-bit Windows, running some tests myself.

Tony

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 12, 2018

From @tonycoz

On Mon, 12 Feb 2018 15​:13​:53 -0800, tonyc wrote​:

On Mon, Feb 12, 2018 at 12​:27​:01PM -0800, Sergey Aleynikov via RT
wrote​:

On Sun, 11 Feb 2018 16​:50​:16 -0800, tonyc wrote​:

Please let me know if this fixes the problem for you.

If it doesn't I'll disable this probing by default.

Now it leaves no traces on linux. But then I've checked windows x64
and x86 builds using VS 2017. x64 passed flawlessly, but for x86 i've
got this​:

- http​://i.imgur.com/9WIXn60.png
- (after pressing close) http​://i.imgur.com/ehVen2N.png

I expected the code I'm using to suppress that error worked on both 32
and 64-bit Windows, running some tests myself.

I'm not seeing failures in the build for x32 on VC 2015, I don't have VC 2017 installed​:

  J​:\dev\perl\git\perl\miniperl.exe "-I..\..\lib" -MExtUtils​::Command -e c
hmod -- 755 ..\..\lib\auto\threads\shared\shared.dll
  ..\perl.exe -I..\lib -I. ..\dist\Storable\stacksize --core
probe for max. stack sizes...
65000 failed, try less 32550 ...
32550 failed, try less 16325 ...
16325 failed, try less 8213 ...
8213 failed, try less 4157 ...
4157 passed, try more 6185 ...
6185 passed, try more 7199 ...
7199 failed, try less 6692 ...
6692 failed, try less 6439 ...
6439 passed, try more 6565 ...
6565 failed, try less 6502 ...
6502 failed, try less 6471 ...
6471 failed, try less 6455 ...
6455 passed, try more 6463 ...
6463 failed, try less 6459 ...
6459 passed, try more 6461 ...
6461 passed, try more 6462 ...
6462 passed, try more 6462 ...
MAX_DEPTH = 6462
3000 passed, try more 3000 ...
MAX_DEPTH_HASH = 3000
  if not exist ..\lib\Storable mkdir ..\lib\Storable
  copy ..\dist\Storable\lib\Storable\Limit.pm ..\lib\Storable\Limit.pm
  1 file(s) copied.

So I don't know what's going on.

I tested​:

J​:\dev\perl\git\perl\win32>git describe
v5.27.8-253-g141513f

I ran the build with​:

  nmake CCTYPE=MSVC140 WIN64=undef test-prep

Tony

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 13, 2018

From @dur-randir

On Mon, 12 Feb 2018 15​:55​:00 -0800, tonyc wrote​:

I'm not seeing failures in the build for x32 on VC 2015, I don't have
VC 2017 installed​:

nmake CCTYPE=MSVC140 WIN64=undef test-prep

My second try to build 32-bit version also succeeded, so I think it's fine. Seems like I've messed with consequent builds.

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 14, 2018

From @tonycoz

On Mon, 12 Feb 2018 16​:04​:20 -0800, randir wrote​:

On Mon, 12 Feb 2018 15​:55​:00 -0800, tonyc wrote​:

I'm not seeing failures in the build for x32 on VC 2015, I don't have
VC 2017 installed​:

nmake CCTYPE=MSVC140 WIN64=undef test-prep

My second try to build 32-bit version also succeeded, so I think it's
fine. Seems like I've messed with consequent builds.

Thanks, closing.

Tony

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 14, 2018

@tonycoz - Status changed from 'open' to 'pending release'

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 21, 2018

From @skaji

It is true that the perl build does not leave core files anymore,
but it still emits "Segmentation fault (core dumped)" message on my environment.
https://gist.github.com/skaji/4fb507113bb7a05a9514e323a001b9a3

It would be nice if it does not emit such messages either.

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 21, 2018

From [Unknown Contact. See original ticket]

It is true that the perl build does not leave core files anymore,
but it still emits "Segmentation fault (core dumped)" message on my environment.
https://gist.github.com/skaji/4fb507113bb7a05a9514e323a001b9a3

It would be nice if it does not emit such messages either.

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 26, 2018

From @tonycoz

On Wed, 21 Feb 2018 08​:13​:29 -0800, skaji@​cpan.org wrote​:

It is true that the perl build does not leave core files anymore,
but it still emits "Segmentation fault (core dumped)" message on my
environment.
https://gist.github.com/skaji/4fb507113bb7a05a9514e323a001b9a3

It would be nice if it does not emit such messages either.

That should be fixed by 3d79e57.

Tony

@p5pRT
Copy link
Author

@p5pRT p5pRT commented May 6, 2018

From shoichikaji@gmail.com

Thanks Tony.

On the other hand, I also found that "segfault" messages might be emitted
to syslog.
Please look at
https://gist.github.com/skaji/1d7a5d57da120e8ca6c2807b4e1a0307

Because people often monitor syslog data and alert themselves when a log
pattern is detected,
I think we should stop these messages if possible (but I don't know how).
On Mon, Feb 26, 2018 at 4​:11 PM Tony Cook via RT <perlbug-followup@​perl.org>
wrote​:

On Wed, 21 Feb 2018 08​:13​:29 -0800, skaji@​cpan.org wrote​:

It is true that the perl build does not leave core files anymore,
but it still emits "Segmentation fault (core dumped)" message on my
environment.
https://gist.github.com/skaji/4fb507113bb7a05a9514e323a001b9a3

It would be nice if it does not emit such messages either.

That should be fixed by 3d79e57.

Tony

---
via perlbug​: queue​: perl5 status​: pending release
https://rt-archive.perl.org/perl5/Ticket/Display.html?id=132849

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Jun 23, 2018

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release yesterday of Perl 5.28.0, this and 185 other issues have been
resolved.

Perl 5.28.0 may be downloaded via​:
https://metacpan.org/release/XSAWYERX/perl-5.28.0

If you find that the problem persists, feel free to reopen this ticket.

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Jun 23, 2018

@khwilliamson - Status changed from 'pending release' to 'resolved'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant