Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"panic: free from wrong pool" while freeing the env after fork on darwin #14955

Closed
p5pRT opened this issue Oct 1, 2015 · 24 comments
Closed

"panic: free from wrong pool" while freeing the env after fork on darwin #14955

p5pRT opened this issue Oct 1, 2015 · 24 comments

Comments

@p5pRT
Copy link
Collaborator

@p5pRT p5pRT commented Oct 1, 2015

Migrated from rt.perl.org#126240 (status was 'resolved')

Searchable as RT126240$

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 1, 2015

From perl@profvince.com

Created by perl@profvince.com

fork() causes a panic at destruction time with a debugging threaded
blead on Mac OS X Yosemite :

$ ./perl -e '$pid = fork and waitpid $pid, 0'
panic​: free from wrong pool, 0!=7f8828801200 during global destruction.

which also causes t/op/magic.t to fail.

valgrind says :

$ valgrind ./perl -e '$pid = fork and waitpid $pid, 0'
==9987== Memcheck, a memory error detector
==9987== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==9987== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==9987== Command​: ./perl -e $pid\ =\ fork\ and\ waitpid\ $pid,\ 0
==9987== ==9988== Invalid read of size 8
==9988== at 0x1001E76FC​: Perl_safesysfree (util.c​:348)
==9988== by 0x100065E23​: perl_destruct (perl.c​:836)
==9988== by 0x100000AF1​: main (perlmain.c​:127)
==9988== Address 0x100e5c038 is 24 bytes before a block of size 19 alloc'd
==9988== at 0x100627EBB​: malloc (in
/usr/local/Cellar/valgrind/3.11.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==9988== by 0x10081CAD5​: __setenv (in /usr/lib/system/libsystem_c.dylib)
==9988== by 0x10081CE59​: setenv (in /usr/lib/system/libsystem_c.dylib)
==9988== by 0x100A7B4A9​: xpc_atfork_child (in
/usr/lib/system/libxpc.dylib)
==9988== by 0x100630B9E​: libSystem_atfork_child (in
/usr/lib/libSystem.B.dylib)
==9988== by 0x1007C8F07​: fork (in /usr/lib/system/libsystem_c.dylib)
==9988== by 0x1001F234C​: Perl_my_fork (util.c​:2705)
==9988== by 0x10042636C​: Perl_pp_fork (pp_sys.c​:4189)
==9988== by 0x1001E41BB​: Perl_runops_debug (dump.c​:2224)
==9988== by 0x10006F9CA​: S_run_body (perl.c​:2456)
==9988== by 0x10006E89A​: perl_run (perl.c​:2379)
==9988== by 0x100000A61​: main (perlmain.c​:116)
==9988== ==9988== Invalid read of size 8
==9988== at 0x1001E7714​: Perl_safesysfree (util.c​:350)
==9988== by 0x100065E23​: perl_destruct (perl.c​:836)
==9988== by 0x100000AF1​: main (perlmain.c​:127)
==9988== Address 0x100e5c038 is 24 bytes before a block of size 19 alloc'd
==9988== at 0x100627EBB​: malloc (in
/usr/local/Cellar/valgrind/3.11.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==9988== by 0x10081CAD5​: __setenv (in /usr/lib/system/libsystem_c.dylib)
==9988== by 0x10081CE59​: setenv (in /usr/lib/system/libsystem_c.dylib)
==9988== by 0x100A7B4A9​: xpc_atfork_child (in
/usr/lib/system/libxpc.dylib)
==9988== by 0x100630B9E​: libSystem_atfork_child (in
/usr/lib/libSystem.B.dylib)
==9988== by 0x1007C8F07​: fork (in /usr/lib/system/libsystem_c.dylib)
==9988== by 0x1001F234C​: Perl_my_fork (util.c​:2705)
==9988== by 0x10042636C​: Perl_pp_fork (pp_sys.c​:4189)
==9988== by 0x1001E41BB​: Perl_runops_debug (dump.c​:2224)
==9988== by 0x10006F9CA​: S_run_body (perl.c​:2456)
==9988== by 0x10006E89A​: perl_run (perl.c​:2379)
==9988== by 0x100000A61​: main (perlmain.c​:116)
==9988== panic​: free from wrong pool, 0!=100e27150 during global
destruction.
==9988== ==9988== HEAP SUMMARY​:
==9988== in use at exit​: 186,142 bytes in 1,021 blocks
==9988== total heap usage​: 1,209 allocs, 188 frees, 212,558 bytes
allocated
==9988== ==9988== LEAK SUMMARY​:
==9988== definitely lost​: 231 bytes in 1 blocks
==9988== indirectly lost​: 0 bytes in 0 blocks
==9988== possibly lost​: 0 bytes in 0 blocks
==9988== still reachable​: 150,107 bytes in 591 blocks
==9988== suppressed​: 35,804 bytes in 429 blocks
==9988== Rerun with --leak-check=full to see details of leaked memory
==9988== ==9988== For counts of detected and suppressed errors, rerun
with​: -v
==9988== ERROR SUMMARY​: 2 errors from 2 contexts (suppressed​: 0 from 0)
==9987== ==9987== HEAP SUMMARY​:
==9987== in use at exit​: 34,920 bytes in 427 blocks
==9987== total heap usage​: 1,199 allocs, 772 frees, 202,849 bytes
allocated
==9987== ==9987== LEAK SUMMARY​:
==9987== definitely lost​: 231 bytes in 1 blocks
==9987== indirectly lost​: 0 bytes in 0 blocks
==9987== possibly lost​: 0 bytes in 0 blocks
==9987== still reachable​: 0 bytes in 0 blocks
==9987== suppressed​: 34,689 bytes in 426 blocks
==9987== Rerun with --leak-check=full to see details of leaked memory
==9987== ==9987== For counts of detected and suppressed errors, rerun
with​: -v
==9987== ERROR SUMMARY​: 0 errors from 0 contexts (suppressed​: 0 from 0)

Perl Info

Flags:
     category=core
     severity=medium

Site configuration information for perl 5.23.4:

Configured by vincent at Thu Oct  1 13:59:15 BRT 2015.

Summary of my perl5 (revision 5 version 23 subversion 4) configuration:
   Commit id: 4b89cb47a04366ffd503d523f2fcf2375cec0a8e
   Platform:
     osname=darwin, osvers=14.5.0, archname=darwin-thread-multi-2level
     uname='darwin dhcp-147-65-161-189.impa.br 14.5.0 darwin kernel 
version 14.5.0: wed jul 29 02:26:53 pdt 2015; 
root:xnu-2782.40.9~1release_x86_64 x86_64 '
     config_args='-des -Dusedevel -Doptimize=-O0 -g3 -DDEBUGGING 
-Duseithreads'
     hint=recommended, useposix=true, d_sigaction=define
     useithreads=define, usemultiplicity=define
     use64bitint=define, use64bitall=define, uselongdouble=undef
     usemymalloc=n, bincompat5005=undef
   Compiler:
     cc='cc', ccflags ='-fno-common -DPERL_DARWIN -DDEBUGGING 
-fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include',
     optimize='-O0 -g3',
     cppflags='-fno-common -DPERL_DARWIN -DDEBUGGING 
-fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
     ccversion='', gccversion='4.2.1 Compatible Apple LLVM 7.0.0 
(clang-700.0.72)', gccosandvers=''
     intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, 
doublekind=3
     d_longlong=define, longlongsize=8, d_longdbl=define, 
longdblsize=16, longdblkind=3
     ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', 
lseeksize=8
     alignbytes=8, prototype=define
   Linker and Libraries:
     ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc', ldflags =' 
-fstack-protector-strong -L/usr/local/lib'
     libpth=/usr/local/lib 
/Library/Developer/CommandLineTools/usr/bin/../lib/clang/7.0.0/lib 
/Library/Developer/CommandLineTools/usr/lib /usr/lib
     libs=-lpthread -lgdbm -ldbm -ldl -lm -lutil -lc
     perllibs=-lpthread -ldl -lm -lutil -lc
     libc=, so=dylib, useshrplib=false, libperl=libperl.a
     gnulibc_version=''
   Dynamic Linking:
     dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
     cccdlflags=' ', lddlflags=' -bundle -undefined dynamic_lookup 
-L/usr/local/lib -fstack-protector-strong'



@INC for perl 5.23.4:
     /usr/local/lib/perl5/site_perl/5.23.4/darwin-thread-multi-2level
     /usr/local/lib/perl5/site_perl/5.23.4
     /usr/local/lib/perl5/5.23.4/darwin-thread-multi-2level
     /usr/local/lib/perl5/5.23.4
     /usr/local/lib/perl5/site_perl
     .
     /tmp/perl/lib


Environment for perl 5.23.4:
     DYLD_LIBRARY_PATH=/tmp/perl
     HOME=/Users/vincent
     LANG=C
     LANGUAGE (unset)
     LC_ALL=C
     LD_LIBRARY_PATH=/usr/local/lib:/usr/lib
     LOGDIR (unset)
 
PATH=/Users/vincent/bin:/Users/vincent/perl/builds/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/texbin
     PERL5LIB=
     PERLDOC_PAGER=less -R
     PERL_BADLANG (unset)
     SHELL=/bin/bash

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 6, 2015

From @bulk88

On Thu Oct 01 10​:20​:48 2015, perl@​profvince.com wrote​:

This is a bug report for perl from perl@​profvince.com,
generated with the help of perlbug 1.40 running under perl 5.23.4.

-----------------------------------------------------------------
[Please describe your issue here]

fork() causes a panic at destruction time with a debugging threaded
blead on Mac OS X Yosemite :

$ ./perl -e '$pid = fork and waitpid $pid, 0'
panic​: free from wrong pool, 0!=7f8828801200 during global
destruction.

which also causes t/op/magic.t to fail.

valgrind says :

$ valgrind ./perl -e '$pid = fork and waitpid $pid, 0'
==9987== Memcheck, a memory error detector
==9987== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et
al.
==9987== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright
info
==9987== Command​: ./perl -e $pid\ =\ fork\ and\ waitpid\ $pid,\ 0
==9987== ==9988== Invalid read of size 8
==9988== at 0x1001E76FC​: Perl_safesysfree (util.c​:348)
==9988== by 0x100065E23​: perl_destruct (perl.c​:836)

What is exactly is perl.c​:836 ? What C code is at line 836 on your perl?

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 6, 2015

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 6, 2015

From perl@profvince.com

What is exactly is perl.c​:836 ? What C code is at line 836 on your perl?

gitweb is kind enough to answer your question :

http​://perl5.git.perl.org/perl.git/blob/4b89cb47a04366ffd503d523f2fcf2375cec0a8e​:/perl.c#l836

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 9, 2015

From @tonycoz

On Thu Oct 01 10​:20​:48 2015, perl@​profvince.com wrote​:

fork() causes a panic at destruction time with a debugging threaded
blead on Mac OS X Yosemite :

$ ./perl -e '$pid = fork and waitpid $pid, 0'
panic​: free from wrong pool, 0!=7f8828801200 during global
destruction.

I'm not seeing this in blead​:

neptune​:perl tony$ uname -a
Darwin neptune.local 14.5.0 Darwin Kernel Version 14.5.0​: Tue Sep 1 21​:23​:09 PDT 2015; root​:xnu-2782.50.1~1/RELEASE_X86_64 x86_64
neptune​:perl tony$ ./perl -v

This is perl 5, version 23, subversion 4 (v5.23.4 (v5.23.3-83-gf7294c1)) built for darwin-thread-multi-2level

...
neptune​:perl tony$ ./perl -e '$pid = fork and waitpid $pid, 0'
neptune​:perl tony$ ./perl -Ilib -V​:config_args
config_args='-des -Dusedevel -Doptimize=-O0 -g3 -DDEBUGGING -Duseithreads';
neptune​:perl tony$

Tony

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 9, 2015

From perl@profvince.com

Le 09/11/2015 21​:23, Tony Cook via RT a écrit :

On Thu Oct 01 10​:20​:48 2015, perl@​profvince.com wrote​:

fork() causes a panic at destruction time with a debugging threaded
blead on Mac OS X Yosemite :

$ ./perl -e '$pid = fork and waitpid $pid, 0'
panic​: free from wrong pool, 0!=7f8828801200 during global
destruction.

I'm not seeing this in blead​:

neptune​:perl tony$ uname -a
Darwin neptune.local 14.5.0 Darwin Kernel Version 14.5.0​: Tue Sep 1 21​:23​:09 PDT 2015; root​:xnu-2782.50.1~1/RELEASE_X86_64 x86_64
neptune​:perl tony$ ./perl -v

This is perl 5, version 23, subversion 4 (v5.23.4 (v5.23.3-83-gf7294c1)) built for darwin-thread-multi-2level

...
neptune​:perl tony$ ./perl -e '$pid = fork and waitpid $pid, 0'
neptune​:perl tony$ ./perl -Ilib -V​:config_args
config_args='-des -Dusedevel -Doptimize=-O0 -g3 -DDEBUGGING -Duseithreads';
neptune​:perl tony$

Well I still do :

$ ./perl -Ilib -e'$pid = fork and waitpid $pid, 0'
panic​: free from wrong pool, 0!=7f9ae1001200 during global destruction.

$ uname -a
Darwin vega.local 14.5.0 Darwin Kernel Version 14.5.0​: Tue Sep 1
21​:23​:09 PDT 2015; root​:xnu-2782.50.1~1/RELEASE_X86_64 x86_64

$ ./perl -Ilib -V
Summary of my perl5 (revision 5 version 23 subversion 5) configuration​:
  Commit id​: a74a9a4
  Platform​:
  osname=darwin, osvers=14.5.0, archname=darwin-thread-multi-2level
  uname='darwin vega.local 14.5.0 darwin kernel version 14.5.0​: tue
sep 1 21​:23​:09 pdt 2015; root​:xnu-2782.50.1~1release_x86_64 x86_64 '
  config_args='-des -Dusedevel -Doptimize=-O0 -g3 -DDEBUGGING
-Duseithreads'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=define, usemultiplicity=define
  use64bitint=define, use64bitall=define, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-fno-common -DPERL_DARWIN
-mmacosx-version-min=10.10 -DDEBUGGING -fno-strict-aliasing -pipe
-fstack-protector-strong -I/usr/local/include',
  optimize='-O0 -g3',
  cppflags='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.10
-DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong
-I/usr/local/include'
  ccversion='', gccversion='4.2.1 Compatible Apple LLVM 7.0.0
(clang-700.1.76)', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678,
doublekind=3
  d_longlong=define, longlongsize=8, d_longdbl=define,
longdblsize=16, longdblkind=3
  ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -mmacosx-version-min=10.10
-fstack-protector-strong -L/usr/local/lib'
  libpth=/usr/local/lib
/Library/Developer/CommandLineTools/usr/bin/../lib/clang/7.0.0/lib
/Library/Developer/CommandLineTools/usr/lib /usr/lib
  libs=-lpthread -lgdbm -ldbm -ldl -lm -lutil -lc
  perllibs=-lpthread -ldl -lm -lutil -lc
  libc=, so=dylib, useshrplib=false, libperl=libperl.a
  gnulibc_version=''
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
  cccdlflags=' ', lddlflags=' -mmacosx-version-min=10.10 -bundle
-undefined dynamic_lookup -L/usr/local/lib -fstack-protector-strong'

Characteristics of this binary (from libperl)​:
  Compile-time options​: DEBUGGING HAS_TIMES MULTIPLICITY PERLIO_LAYERS
  PERL_COPY_ON_WRITE PERL_DONT_CREATE_GVSV
  PERL_HASH_FUNC_ONE_AT_A_TIME_HARD
  PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP
  PERL_PRESERVE_IVUV PERL_TRACK_MEMPOOL
PERL_USE_DEVEL
  USE_64_BIT_ALL USE_64_BIT_INT USE_ITHREADS
  USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE
  USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME
  USE_PERLIO USE_PERL_ATOF USE_REENTRANT_API
  Built under darwin
  Compiled at Nov 9 2015 21​:35​:14
  %ENV​:
  PERLDOC_PAGER="less -R"
  @​INC​:
  lib
  /usr/local/lib/perl5/site_perl/5.23.5/darwin-thread-multi-2level
  /usr/local/lib/perl5/site_perl/5.23.5
  /usr/local/lib/perl5/5.23.5/darwin-thread-multi-2level
  /usr/local/lib/perl5/5.23.5
  /usr/local/lib/perl5/site_perl
  .

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Dec 8, 2015

From @tonycoz

On Mon Nov 09 15​:42​:47 2015, perl@​profvince.com wrote​:

Le 09/11/2015 21​:23, Tony Cook via RT a écrit :

On Thu Oct 01 10​:20​:48 2015, perl@​profvince.com wrote​:

fork() causes a panic at destruction time with a debugging threaded
blead on Mac OS X Yosemite :

$ ./perl -e '$pid = fork and waitpid $pid, 0'
panic​: free from wrong pool, 0!=7f8828801200 during global
destruction.

I'm not seeing this in blead​:

Well I still do :

$ ./perl -Ilib -e'$pid = fork and waitpid $pid, 0'
panic​: free from wrong pool, 0!=7f9ae1001200 during global
destruction.

I've now reproduced this.

It doesn't happen within a ssh shell, but it does in Terminal.app.

Tony

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Dec 8, 2015

From @tonycoz

On Mon Dec 07 20​:46​:09 2015, tonyc wrote​:

I've now reproduced this.

It doesn't happen within a ssh shell, but it does in Terminal.app.

What I suspect is happening is the xpc_atfork_child function is calling setenv() and libc is rearranging the environment, invalidating our environ pointers (allocated with malloc() instead of safesysmalloc()).

Building with -DPERL_USE_SAFE_PUTENV prevents the errors.

Tony

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Dec 8, 2015

From perl@profvince.com

Le 08/12/2015 03​:54, Tony Cook via RT a écrit :

On Mon Dec 07 20​:46​:09 2015, tonyc wrote​:

I've now reproduced this.

It doesn't happen within a ssh shell, but it does in Terminal.app.

What I suspect is happening is the xpc_atfork_child function is calling setenv() and libc is rearranging the environment, invalidating our environ pointers (allocated with malloc() instead of safesysmalloc()).

Building with -DPERL_USE_SAFE_PUTENV prevents the errors.

Tony

This makes sense and matches what I've been seeing intermittently from
valgrind. Does this mean that this option should always be enabled on
darwin then? What are the trade-offs?

Vincent

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Dec 8, 2015

From @tonycoz

On Tue Dec 08 03​:34​:17 2015, perl@​profvince.com wrote​:

Le 08/12/2015 03​:54, Tony Cook via RT a écrit :

Building with -DPERL_USE_SAFE_PUTENV prevents the errors.

Tony

This makes sense and matches what I've been seeing intermittently from
valgrind. Does this mean that this option should always be enabled on
darwin then? What are the trade-offs?

I suspect that we should be using -DPERL_USE_SAFE_PUTENV on any system we can, since it prevents conflicts with external libraries that use setenv()/unsetenv(), which was the problem here.

From looking at the hints, riscos, sufficiently modern solaris and cygwin enable it.

Please try the attached patch.

Tony

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Dec 8, 2015

From @tonycoz

0001-perl-126240-use-DPERL_USE_SAFE_PUTENV-where-possible.patch
From 1bde2c639b40d1021477ad7b48c14805b32c7b60 Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Wed, 9 Dec 2015 10:20:46 +1100
Subject: [perl #126240] use -DPERL_USE_SAFE_PUTENV where possible on OS X

On threaded builds on OS X, libSystem registers atfork handlers that
call setenv(), which internally modifies members of environ[], setting
them to malloc()ed blocks.

In some cases Perl_my_setenv() reallocates environ[] using
safesysmalloc(), which under debugging builds adds a tracking header,
and if perl_destruct() sees that environ[] has been reallocated, frees
it with safesysfree().

When these combine, perl attempts to free the malloc()ed block with
safesysfree(), which attempts to access the tracking header, causing
an invalid access in tools like valgrind, or a "free from wrong pool"
error, since the header contains unrelated data.

Avoid this mess by letting libc manage environ[] if unsetenv() is
available.
---
 hints/darwin.sh | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/hints/darwin.sh b/hints/darwin.sh
index ee20760..2af6ae7 100644
--- a/hints/darwin.sh
+++ b/hints/darwin.sh
@@ -490,3 +490,13 @@ esac
 # makefile in the same place.  Since Darwin uses GNU make, this dodges
 # the problem.
 firstmakefile=GNUmakefile;
+
+# Parts of the system call setenv(), in particular in an atfork handler.
+# This causes problems when the child tries to clean up environ[], so
+# let libc manage environ[].
+cat >> config.over <<'EOOVER'
+if test "$d_unsetenv" = "$define" -a \
+    `expr "$ccflags" : '.*-DPERL_USE_SAFE_PUTENV'` -eq 0; then
+        ccflags="$ccflags -DPERL_USE_SAFE_PUTENV"
+fi
+EOOVER
-- 
2.4.9 (Apple Git-60)

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Dec 9, 2015

From @jkeenan

On Tue Dec 08 15​:38​:01 2015, tonyc wrote​:

On Tue Dec 08 03​:34​:17 2015, perl@​profvince.com wrote​:

Le 08/12/2015 03​:54, Tony Cook via RT a écrit :

Building with -DPERL_USE_SAFE_PUTENV prevents the errors.

Tony

This makes sense and matches what I've been seeing intermittently
from
valgrind. Does this mean that this option should always be enabled on
darwin then? What are the trade-offs?

I suspect that we should be using -DPERL_USE_SAFE_PUTENV on any system
we can, since it prevents conflicts with external libraries that use
setenv()/unsetenv(), which was the problem here.

From looking at the hints, riscos, sufficiently modern solaris and
cygwin enable it.

Please try the attached patch.

Tony

Smoking in branch​: smoke-me/jkeenan/126240-darwin-use-safe-putenv

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Dec 9, 2015

From @rjbs

This patch cleared up my tests failures.

--
rjbs

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Dec 9, 2015

From @jkeenan

On Tue Dec 08 16​:22​:42 2015, jkeenan wrote​:

On Tue Dec 08 15​:38​:01 2015, tonyc wrote​:

On Tue Dec 08 03​:34​:17 2015, perl@​profvince.com wrote​:

Le 08/12/2015 03​:54, Tony Cook via RT a écrit :

Building with -DPERL_USE_SAFE_PUTENV prevents the errors.

Tony

This makes sense and matches what I've been seeing intermittently
from
valgrind. Does this mean that this option should always be enabled on
darwin then? What are the trade-offs?

I suspect that we should be using -DPERL_USE_SAFE_PUTENV on any system
we can, since it prevents conflicts with external libraries that use
setenv()/unsetenv(), which was the problem here.

From looking at the hints, riscos, sufficiently modern solaris and
cygwin enable it.

Please try the attached patch.

Tony

Smoking in branch​: smoke-me/jkeenan/126240-darwin-use-safe-putenv

Built successfully on older Darwin; see attachment.

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Dec 9, 2015

From @jkeenan

Summary of my perl5 (revision 5 version 23 subversion 6) configuration​:
  Commit id​: 35abf35
  Platform​:
  osname=darwin, osvers=8.11.0, archname=darwin-2level
  uname='darwin macintosh-9.local 8.11.0 darwin kernel version 8.11.0​: wed oct 10 18​:26​:00 pdt 2007; root​:xnu-792.24.17~1release_ppc power macintosh powerpc '
  config_args='-des -Dusedevel'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  use64bitint=undef, use64bitall=undef, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-fno-common -DPERL_DARWIN -fno-strict-aliasing -pipe -I/usr/local/include -I/opt/local/include -D_FORTIFY_SOURCE=2 -DPERL_USE_SAFE_PUTENV',
  optimize='-O3',
  cppflags='-fno-common -DPERL_DARWIN -fno-strict-aliasing -pipe -I/usr/local/include -I/opt/local/include'
  ccversion='', gccversion='4.0.1 (Apple Computer, Inc. build 5250)', gccosandvers=''
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321, doublekind=4
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=6
  ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc', ldflags =' -L/usr/local/lib -L/opt/local/lib'
  libpth=/usr/local/lib /usr/lib /opt/local/lib
  libs=-lpthread -ldbm -ldl -lm -lc
  perllibs=-lpthread -ldl -lm -lc
  libc=, so=dylib, useshrplib=false, libperl=libperl.a
  gnulibc_version=''
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
  cccdlflags=' ', lddlflags=' -bundle -undefined dynamic_lookup -L/usr/local/lib -L/opt/local/lib'

Characteristics of this binary (from libperl)​:
  Compile-time options​: HAS_TIMES PERLIO_LAYERS PERL_COPY_ON_WRITE
  PERL_DONT_CREATE_GVSV
  PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP
  PERL_PRESERVE_IVUV PERL_USE_DEVEL
  PERL_USE_SAFE_PUTENV USE_LARGE_FILES USE_LOCALE
  USE_LOCALE_COLLATE USE_LOCALE_CTYPE
  USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO
  USE_PERL_ATOF
  Built under darwin
  Compiled at Dec 8 2015 20​:13​:46
  %ENV​:
  PERLBREW_BASHRC_VERSION="0.59"
  PERLBREW_HOME="/Users/jimk/.perlbrew"
  PERLBREW_ROOT="/Users/jimk/perl5/perlbrew"
  PERL_WORKDIR="gitwork/perl"
  @​INC​:
  lib
  /usr/local/lib/perl5/site_perl/5.23.6/darwin-2level
  /usr/local/lib/perl5/site_perl/5.23.6
  /usr/local/lib/perl5/5.23.6/darwin-2level
  /usr/local/lib/perl5/5.23.6
  /usr/local/lib/perl5/site_perl
  .

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Dec 9, 2015

From @iabyn

On Tue, Dec 08, 2015 at 03​:38​:02PM -0800, Tony Cook via RT wrote​:

On Tue Dec 08 03​:34​:17 2015, perl@​profvince.com wrote​:

Le 08/12/2015 03​:54, Tony Cook via RT a écrit :

Building with -DPERL_USE_SAFE_PUTENV prevents the errors.

Tony

This makes sense and matches what I've been seeing intermittently from
valgrind. Does this mean that this option should always be enabled on
darwin then? What are the trade-offs?

I suspect that we should be using -DPERL_USE_SAFE_PUTENV on any system we can

By "can", do you mean any system that physically works with
PERL_USE_SAFE_PUTENV, or any system that doesn't leak with
PERL_USE_SAFE_PUTENV?

--
"Procrastination grows to fill the available time"
  -- Mitchell's corollary to Parkinson's Law

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Dec 16, 2015

From @tonycoz

On Wed Dec 09 07​:08​:21 2015, davem wrote​:

On Tue, Dec 08, 2015 at 03​:38​:02PM -0800, Tony Cook via RT wrote​:

I suspect that we should be using -DPERL_USE_SAFE_PUTENV on any
system we can

By "can", do you mean any system that physically works with
PERL_USE_SAFE_PUTENV, or any system that doesn't leak with
PERL_USE_SAFE_PUTENV?

I guess that's the hard bit to define.

I'm guessing any system that has unsetenv() will be safe, since setenv(), getenv() and unsetenv() *should* be working together to manage memory correctly.

putenv()'s memory handling isn't well defined in POSIX, ie. there's no
assumption in POSIX[1] (nor the man pages for Linux, BSD, OS X that I
saw) that such memory is allocated with malloc(), so an implementation
can't really free() it safely.

POSIX also mentions[2] that using putenv() can (theoretically) slow down getenv() if an implementation uses an internal structure to speed up environment variable lookups.

Tony

[1] http​://pubs.opengroup.org/onlinepubs/9699919799/functions/putenv.html
The rationale talks about free() in passing, but there's no requirement

[2] http​://pubs.opengroup.org/onlinepubs/9699919799/functions/getenv.html
in the rationale.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Dec 17, 2015

From @tonycoz

On Tue Dec 08 17​:50​:39 2015, rjbs wrote​:

This patch cleared up my tests failures.

Applied as e396210.

Leaving this open a little longer in case there's more discussion of -DPERL_USE_SAFE_PUTENV in the more general case.

Tony

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jan 7, 2016

From @tonycoz

On Wed Dec 16 22​:06​:58 2015, tonyc wrote​:

On Tue Dec 08 17​:50​:39 2015, rjbs wrote​:

This patch cleared up my tests failures.

Applied as e396210.

Leaving this open a little longer in case there's more discussion of
-DPERL_USE_SAFE_PUTENV in the more general case.

This required an extra fix in 12ffbb1.

Tony

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented May 7, 2016

From mojca.miklavec.lists@gmail.com

I'm having problems building Perl 5.24.0-RC5 (I didn't try older ones), see https​://rt.perl.org/Ticket/Display.html?id=128093 and https://trac.macports.org/ticket/51330

I'm not saying that commits related to this ticket lead to a build problem, but this is the only ticket related to "environ" that I found in the changelog.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 22, 2017

From @tonycoz

On Sat, 07 May 2016 04​:55​:20 -0700, mojca.miklavec.lists@​gmail.com wrote​:

I'm having problems building Perl 5.24.0-RC5 (I didn't try older
ones), see https://rt-archive.perl.org/perl5/Ticket/Display.html?id=128093 and
https://trac.macports.org/ticket/51330

I'm not saying that commits related to this ticket lead to a build
problem, but this is the only ticket related to "environ" that I found
in the changelog.

This was unrelated, closing.

Tony

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 22, 2017

@tonycoz - Status changed from 'open' to 'pending release'

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented May 30, 2017

From @khwilliamson

Thank you for filing this report. You have helped make Perl better.

With the release today of Perl 5.26.0, this and 210 other issues have been
resolved.

Perl 5.26.0 may be downloaded via​:
https://metacpan.org/release/XSAWYERX/perl-5.26.0

If you find that the problem persists, feel free to reopen this ticket.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented May 30, 2017

@khwilliamson - Status changed from 'pending release' to 'resolved'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.