Out of memory when doing IO via pipe and receive signal

[p5pRT]

Migrated from rt.perl.org#134035 (status was 'open')

Searchable as RT134035$

[p5pRT]

From azrlew@gmail.com

This is a bug report for perl from azrlew@​gmail.com,
generated with the help of perlbug 1.40 running under perl 5.26.1.

Out of memory error (ENOMEN) can occur when Perl is doing IO via pipe and a signal handler is triggered.

Test codes to reproduce the problem can be found here​:
https://github.com/azrle/perl-signal-io-bug

Perl will fail to run at least one of 'perlio-signal-while-safe.pl' and 'perlio-signal-while.pl'.

Serveral Perl versions are affected including 5.16.1, 5.18.2, 5.26.1.
Note that not all versions failed on both.

---

Some crash logs (perl 5.18.2, macos)​:

# perl perlio-signal-while.pl
parent 43396
timeout
perl(43396,0x1173c15c0) malloc​: can't allocate region
*** mach_vm_map(size=18446744073683324928) failed (error code=3)
perl(43396,0x1173c15c0) malloc​: *** set a breakpoint in malloc_error_break to debug
Out of memory!

---

Some gdb results (perl 5.16.1, linux)​:

(gdb) run perlio-signal-while.pl
Starting program​: /usr/bin/perl perlio-signal-while.pl

Breakpoint 1, Perl_sv_gets (sv=0x780ca0, fp=0x76ee90, append=0) at sv.c​:7821
7821 sv.c​: No such file or directory.
(gdb) c 18
Will ignore next 17 crossings of breakpoint 1. Continuing.
parent 25688
timeout

Breakpoint 1, Perl_sv_gets (sv=0x780c28, fp=0x76ee70, append=0) at sv.c​:7821
7821 in sv.c
(gdb) bt
#0 Perl_sv_gets (sv=0x780c28, fp=0x76ee70, append=0) at sv.c​:7821
#1 0x000000000049a8d1 in Perl_do_readline () at pp_hot.c​:1675
#2 0x0000000000496193 in Perl_runops_standard () at run.c​:41
#3 0x0000000000434878 in S_run_body (oldscope=<optimized out>) at perl.c​:2402
#4 perl_run (my_perl=<optimized out>) at perl.c​:2320
#5 0x000000000041e10c in main (argc=2, argv=0x7fffffffdff8, env=0x7fffffffe010) at perlmain.c​:120

(gdb) p ptr
$5 = 0x8d5dd1 "2\n"
(gdb) p ptr-1
$6 = 0x8d5dd0 "42\n"

(gdb) p bp
$7 = 0x76e070 ""

(gdb) p ((const char*)((sv)->sv_u.svu_pv))
$8 = 0x77dfe0 ""

# underflow
(gdb) p bp-((const char*)((sv)->sv_u.svu_pv))
$9 = -65392
(gdb) n
7822 in sv.c
(gdb) p bpx
$10 = 18446744073709486224

---

Flags​:
  category=core
  severity=medium

---

Site configuration information for perl 5.26.1​:

Configured by Ubuntu at Mon Nov 19 15​:54​:44 UTC 2018.

Summary of my perl5 (revision 5 version 26 subversion 1) configuration​:

  Platform​:
  osname=linux
  osvers=4.9.0
  archname=x86_64-linux-gnu-thread-multi
  uname='linux localhost 4.9.0 #1 smp debian 4.9.0 x86_64 gnulinux '
  config_args='-Dusethreads -Duselargefiles -Dcc=x86_64-linux-gnu-gcc -Dcpp=x86_64-linux-gnu-cpp -Dld=x86_64-linux-gnu-gcc -Dccflags=-DDEBIAN -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fdebug-prefix-map=/build/perl-hx1dVS/perl-5.26.1=. -fstack-protector-strong -Wformat -Werror=format-security -Dldflags= -Wl,-Bsymbolic-functions -Wl,-z,relro -Dlddlflags=-shared -Wl,-Bsymbolic-functions -Wl,-z,relro -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.26 -Darchlib=/usr/lib/x86_64-linux-gnu/perl/5.26 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/x86_64-linux-gnu/perl5/5.26 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.26.1 -Dsitearch=/usr/local/lib/x86_64-linux-gnu/perl/5.26.1 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Duse64bitint
-Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -Ui_libutil -Ui_xlocale -Uversiononly -DDEBUGGING=-g -Doptimize=-O2 -dEs -Duseshrplib -Dlibperl=libperl.so.5.26.1'
  hint=recommended
  useposix=true
  d_sigaction=define
  useithreads=define
  usemultiplicity=define
  use64bitint=define
  use64bitall=define
  uselongdouble=undef
  usemymalloc=n
  default_inc_excludes_dot=define
  bincompat5005=undef
  Compiler​:
  cc='x86_64-linux-gnu-gcc'
  ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
  optimize='-O2 -g'
  cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include'
  ccversion=''
  gccversion='7.3.0'
  gccosandvers=''
  intsize=4
  longsize=8
  ptrsize=8
  doublesize=8
  byteorder=12345678
  doublekind=3
  d_longlong=define
  longlongsize=8
  d_longdbl=define
  longdblsize=16
  longdblkind=3
  ivtype='long'
  ivsize=8
  nvtype='double'
  nvsize=8
  Off_t='off_t'
  lseeksize=8
  alignbytes=8
  prototype=define
  Linker and Libraries​:
  ld='x86_64-linux-gnu-gcc'
  ldflags =' -fstack-protector-strong -L/usr/local/lib'
  libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/7/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
  libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
  perllibs=-ldl -lm -lpthread -lc -lcrypt
  libc=libc-2.27.so
  so=so
  useshrplib=true
  libperl=libperl.so.5.26
  gnulibc_version='2.27'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs
  dlext=so
  d_dlsymun=undef
  ccdlflags='-Wl,-E'
  cccdlflags='-fPIC'
  lddlflags='-shared -L/usr/local/lib -fstack-protector-strong'

Locally applied patches​:
  DEBPKG​:debian/cpan_definstalldirs - Provide a sensible INSTALLDIRS default for modules installed from CPAN.
  DEBPKG​:debian/db_file_ver - https://bugs.debian.org/340047 Remove overly restrictive DB_File version check.
  DEBPKG​:debian/doc_info - Replace generic man(1) instructions with Debian-specific information.
  DEBPKG​:debian/enc2xs_inc - https://bugs.debian.org/290336 Tweak enc2xs to follow symlinks and ignore missing @​INC directories.
  DEBPKG​:debian/errno_ver - https://bugs.debian.org/343351 Remove Errno version check due to upgrade problems with long-running processes.
  DEBPKG​:debian/libperl_embed_doc - https://bugs.debian.org/186778 Note that libperl-dev package is required for embedded linking
  DEBPKG​:fixes/respect_umask - Respect umask during installation
  DEBPKG​:debian/writable_site_dirs - Set umask approproately for site install directories
  DEBPKG​:debian/extutils_set_libperl_path - EU​:MM​: set location of libperl.a under /usr/lib
  DEBPKG​:debian/no_packlist_perllocal - Don't install .packlist or perllocal.pod for perl or vendor
  DEBPKG​:debian/fakeroot - Postpone LD_LIBRARY_PATH evaluation to the binary targets.
  DEBPKG​:debian/instmodsh_doc - Debian policy doesn't install .packlist files for core or vendor.
  DEBPKG​:debian/ld_run_path - Remove standard libs from LD_RUN_PATH as per Debian policy.
  DEBPKG​:debian/libnet_config_path - Set location of libnet.cfg to /etc/perl/Net as /usr may not be writable.
  DEBPKG​:debian/perlivp - https://bugs.debian.org/510895 Make perlivp skip include directories in /usr/local
  DEBPKG​:debian/deprecate-with-apt - https://bugs.debian.org/747628 Point users to Debian packages of deprecated core modules
  DEBPKG​:debian/squelch-locale-warnings - https://bugs.debian.org/508764 Squelch locale warnings in Debian package maintainer scripts
  DEBPKG​:debian/patchlevel - https://bugs.debian.org/567489 List packaged patches for 5.26.1-6ubuntu0.3 in patchlevel.h
  DEBPKG​:fixes/document_makemaker_ccflags - https://bugs.debian.org/628522 [rt.cpan.org #68613] Document that CCFLAGS should include $Config{ccflags}
  DEBPKG​:debian/find_html2text - https://bugs.debian.org/640479 Configure CPAN​::Distribution with correct name of html2text
  DEBPKG​:debian/perl5db-x-terminal-emulator.patch - https://bugs.debian.org/668490 Invoke x-terminal-emulator rather than xterm in perl5db.pl
  DEBPKG​:debian/cpan-missing-site-dirs - https://bugs.debian.org/688842 Fix CPAN​::FirstTime defaults with nonexisting site dirs if a parent is writable
  DEBPKG​:fixes/memoize_storable_nstore - [rt.cpan.org #77790] https://bugs.debian.org/587650 Memoize​::Storable​: respect 'nstore' option not respected
  DEBPKG​:debian/makemaker-pasthru - https://bugs.debian.org/758471 Pass LD settings through to subdirectories
  DEBPKG​:debian/makemaker-manext - https://bugs.debian.org/247370 Make EU​::MakeMaker honour MANnEXT settings in generated manpage headers
  DEBPKG​:debian/kfreebsd-softupdates - https://bugs.debian.org/796798 Work around Debian Bug#796798
  DEBPKG​:fixes/autodie-scope - https://bugs.debian.org/798096 Fix a scoping issue with "no autodie" and the "system" sub
  DEBPKG​:fixes/memoize-pod - [rt.cpan.org #89441] Fix POD errors in Memoize
  DEBPKG​:debian/hurd-softupdates - https://bugs.debian.org/822735 Fix t/op/stat.t failures on hurd
  DEBPKG​:fixes/math_complex_doc_great_circle - https://bugs.debian.org/697567 [rt.cpan.org #114104] Math​::Trig​: clarify definition of great_circle_midpoint
  DEBPKG​:fixes/math_complex_doc_see_also - https://bugs.debian.org/697568 [rt.cpan.org #114105] Math​::Trig​: add missing SEE ALSO
  DEBPKG​:fixes/math_complex_doc_angle_units - https://bugs.debian.org/731505 [rt.cpan.org #114106] Math​::Trig​: document angle units
  DEBPKG​:fixes/cpan_web_link - https://bugs.debian.org/367291 CPAN​: Add link to main CPAN web site
  DEBPKG​:fixes/time_piece_doc - https://bugs.debian.org/817925 Time​::Piece​: Improve documentation for add_months and add_years
  DEBPKG​:fixes/extutils_makemaker_reproducible - https​://bugs.debian.org/835815 https://bugs.debian.org/834190 Make perllocal.pod files reproducible
  DEBPKG​:fixes/file_path_hurd_errno - File-Path​: Fix test failure in Hurd due to hard-coded ENOENT
  DEBPKG​:debian/hppa_op_optimize_workaround - https://bugs.debian.org/838613 Temporarily lower the optimization of op.c on hppa due to gcc-6 problems
  DEBPKG​:debian/installman-utf8 - https://bugs.debian.org/840211 Generate man pages with UTF-8 characters
  DEBPKG​:fixes/file_path_chmod_race - https://bugs.debian.org/863870 [rt.cpan.org #121951] Prevent directory chmod race attack.
  DEBPKG​:fixes/extutils_file_path_compat - Correct the order of tests of chmod(). (#294)
  DEBPKG​:fixes/getopt-long-2 - [rt.cpan.org #120300] Withdraw part of commit 5d9947fb445327c7299d8beb009d609bc70066c0, which tries to implement more GNU getopt_long campatibility. GNU
  DEBPKG​:fixes/getopt-long-3 - provide a default value for optional arguments
  DEBPKG​:fixes/getopt-long-4 - https://bugs.debian.org/864544 [rt.cpan.org #122068] Fix issue #122068.
  DEBPKG​:fixes/test-builder-reset - https://bugs.debian.org/865894 Reset inside subtest maintains parent
  DEBPKG​:debian/hppa_opmini_optimize_workaround - https://bugs.debian.org/869122 Lower the optimization level of opmini.c on hppa
  DEBPKG​:debian/sh4_op_optimize_workaround - https://bugs.debian.org/869373 Also lower the optimization level of op.c and opmini.c on sh4
  DEBPKG​:fixes/json-pp-example - [rt.cpan.org #92793] https://bugs.debian.org/871837 fix RT-92793​: bug in SYNOPSIS
  DEBPKG​:debian/perldoc-pager - https://bugs.debian.org/870340 [rt.cpan.org #120229] Fix perldoc terminal escapes when sensible-pager is less
  DEBPKG​:debian/prune_libs - https://bugs.debian.org/128355 Prune the list of libraries wanted to what we actually need.
  DEBPKG​:debian/configure-regen - https://bugs.debian.org/762638 Regenerate Configure et al. after probe unit changes
  DEBPKG​:fixes/rename-filexp.U-phase1 - regen-configure​: rename filexp.U to filexp_path.U, phase 1
  DEBPKG​:fixes/rename-filexp.U-phase2 - regen-configure​: rename filexp.U to filexp_path.U, phase 2
  DEBPKG​:fixes/packaging_test_skips - Skip various tests if PERL_BUILD_PACKAGING is set
  DEBPKG​:debian/mod_paths - Tweak @​INC ordering for Debian
  DEBPKG​:fixes/encode-alias-regexp - https​://bugs.debian.org/880085 fix dankogai/p5-encode#127
  DEBPKG​:fixes/regex-memory-leak - [910a6a8] https://bugs.debian.org/891196 [perl #132892] perl #132892​: avoid leak by mortalizing temporary copy of pattern
  DEBPKG​:fixes/CVE-2018-6797 - [perl #132227] (perl #132227) restart a node if we change to uni rules within the node and encounter a sharp S
  DEBPKG​:fixes/CVE-2018-6798/pt1 - [perl #132063] Heap buffer overflow
  DEBPKG​:fixes/CVE-2018-6798/pt2 - [perl #132063] 5.26.1​: fix TRIE_READ_CHAR and DECL_TRIE_TYPE to account for non-utf8 target
  DEBPKG​:fixes/CVE-2018-6798/pt3 - [perl #132063] (perl #132063) we should no longer warn for this code
  DEBPKG​:fixes/CVE-2018-6798/pt4 - [perl #132063] utf8.c​: Don't dump malformation past first NUL
  DEBPKG​:fixes/CVE-2018-6913 - [perl #131844] (perl #131844) fix various space calculation issues in pp_pack.c
  DEBPKG​:fixes/CVE-2018-12015.patch - [PATCH] [PATCH] Remove existing files before overwriting them
  DEBPKG​:fixes/CVE-2018-18311.patch - [PATCH] Perl_my_setenv(); handle integer wrap
  DEBPKG​:fixes/CVE-2018-18312.patch - [PATCH 242/242] PATCH​: [perl #133423] for 5.26 maint
  DEBPKG​:fixes/CVE-2018-18313.patch - [PATCH] regcomp.c​: Convert some strchr to memchr
  DEBPKG​:fixes/CVE-2018-18314.patch - [PATCH] fix #131649 - extended charclass can trigger assert

---

@​INC for perl 5.26.1​:
  /etc/perl
  /usr/local/lib/x86_64-linux-gnu/perl/5.26.1
  /usr/local/share/perl/5.26.1
  /usr/lib/x86_64-linux-gnu/perl5/5.26
  /usr/share/perl5
  /usr/lib/x86_64-linux-gnu/perl/5.26
  /usr/share/perl/5.26
  /usr/local/lib/site_perl
  /usr/lib/x86_64-linux-gnu/perl-base

---

Environment for perl 5.26.1​:
  HOME=/root
  LANG (unset)
  LANGUAGE (unset)
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)
  PATH=/root/perl5/perlbrew/bin​:/usr/local/sbin​:/usr/local/bin​:/usr/sbin​:/usr/bin​:/sbin​:/bin
  PERLBREW_HOME=/root/.perlbrew
  PERLBREW_ROOT=/root/perl5/perlbrew
  PERLBREW_SHELLRC_VERSION=0.86
  PERL_BADLANG (unset)
  SHELL=/bin/bash

[p5pRT]

From @tonycoz

On Mon, 15 Apr 2019 21​:18​:40 -0700, azrlew@​gmail.com wrote​:

This is a bug report for perl from azrlew@​gmail.com,
generated with the help of perlbug 1.40 running under perl 5.26.1.

Out of memory error (ENOMEN) can occur when Perl is doing IO via pipe
and a signal handler is triggered.

Test codes to reproduce the problem can be found here​:
https://github.com/azrle/perl-signal-io-bug

Perl will fail to run at least one of 'perlio-signal-while-safe.pl'
and 'perlio-signal-while.pl'.

Serveral Perl versions are affected including 5.16.1, 5.18.2, 5.26.1.
Note that not all versions failed on both.

---
Some crash logs (perl 5.18.2, macos)​:

# perl perlio-signal-while.pl
parent 43396
timeout
perl(43396,0x1173c15c0) malloc​: can't allocate region
*** mach_vm_map(size=18446744073683324928) failed (error code=3)
perl(43396,0x1173c15c0) malloc​: *** set a breakpoint in
malloc_error_break to debug
Out of memory!

---
Some gdb results (perl 5.16.1, linux)​:

(gdb) run perlio-signal-while.pl
Starting program​: /usr/bin/perl perlio-signal-while.pl

Breakpoint 1, Perl_sv_gets (sv=0x780ca0, fp=0x76ee90, append=0) at
sv.c​:7821
7821 sv.c​: No such file or directory.
(gdb) c 18
Will ignore next 17 crossings of breakpoint 1. Continuing.
parent 25688
timeout

Breakpoint 1, Perl_sv_gets (sv=0x780c28, fp=0x76ee70, append=0) at
sv.c​:7821
7821 in sv.c
(gdb) bt
#0 Perl_sv_gets (sv=0x780c28, fp=0x76ee70, append=0) at sv.c​:7821
#1 0x000000000049a8d1 in Perl_do_readline () at pp_hot.c​:1675
#2 0x0000000000496193 in Perl_runops_standard () at run.c​:41
#3 0x0000000000434878 in S_run_body (oldscope=<optimized out>) at
perl.c​:2402
#4 perl_run (my_perl=<optimized out>) at perl.c​:2320
#5 0x000000000041e10c in main (argc=2, argv=0x7fffffffdff8,
env=0x7fffffffe010) at perlmain.c​:120

(gdb) p ptr
$5 = 0x8d5dd1 "2\n"
(gdb) p ptr-1
$6 = 0x8d5dd0 "42\n"

(gdb) p bp
$7 = 0x76e070 ""

(gdb) p ((const char*)((sv)->sv_u.svu_pv))
$8 = 0x77dfe0 ""

# underflow
(gdb) p bp-((const char*)((sv)->sv_u.svu_pv))
$9 = -65392
(gdb) n
7822 in sv.c
(gdb) p bpx
$10 = 18446744073709486224

Yeah, the code isn't expecting the SV, $_ in this case, to be modified from underneath it.

Adding​:

  local $_;

to the beginning of the signal handler also fixes the problem and is probably the sane thing to do in a signal handler anyway.

Having perl do the local would fix it for $_, but it would still be broken for any other SV.

The attached patch fixes it for me.

$ ../perl/perl -I../perl/lib perlio-signal-while-safe.pl
parent 24425
timeout
panic​: realloc, size=18446744073709476200 at perlio-signal-while-safe.pl line 27.

At first I thought it might be a safe signals problem, but that isn't the case.

Tony

[p5pRT]

From @tonycoz

0001-perl-134035-ensure-sv_gets-handles-a-signal-handler-.patch

From fb09f9d42c2557570354463dc2299ca2e7f0fc41 Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Wed, 17 Apr 2019 15:45:15 +1000
Subject: (perl #134035) ensure sv_gets() handles a signal handler modifying sv

At a very basic level at least.

In the ticket cases, a signal handler is modifying (and reallocating
PVX) the sv, while sv_gets() retained a pointer to the inside of the
SV.

This still has some problems, like if the signal handler ends up
shortening SV, there may be old data left between the old position
and the new position, but I think that's a case of user error.
---
 sv.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/sv.c b/sv.c
index 9b659e8c16..99998b1d0e 100644
--- a/sv.c
+++ b/sv.c
@@ -8755,7 +8755,10 @@ Perl_sv_gets(pTHX_ SV *const sv, PerlIO *const fp, I32 append)
 
             Note we have to deal with the char in 'i' if we are not at EOF
         */
+        bpx = bp - (STDCHAR*)SvPVX_const(sv);
+        /* signals might be called here, possibly modifying sv */
 	i   = PerlIO_getc(fp);		/* get more characters */
+        bp = (STDCHAR*)SvPVX_const(sv) + bpx;
 
 	DEBUG_Pv(PerlIO_printf(Perl_debug_log,
 	   "Screamer: post: FILE * thinks ptr=%" UVuf ", cnt=%" IVdf ", base=%" UVuf "\n",
-- 
2.11.0

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'