Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

memory leak when including a file with "use utf8" #13587

Closed
p5pRT opened this issue Feb 7, 2014 · 32 comments
Closed

memory leak when including a file with "use utf8" #13587

p5pRT opened this issue Feb 7, 2014 · 32 comments

Comments

@p5pRT
Copy link

@p5pRT p5pRT commented Feb 7, 2014

Migrated from rt.perl.org#121200 (status was 'resolved')

Searchable as RT121200$

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 7, 2014

From @lavv17

Created by @lavv17

Every time perl includes a file with "use utf8" and text constants
it leaks memory.

Here is my test code​:
---------------------
my $mem = 0;
sub report_memory () {
  my $next_mem = qx[ps -p $$ -o size=]+0;
  printf "%+8d = %8d K\n", $next_mem-$mem, $next_mem;
  $mem = $next_mem;
}

for (0..1e5) {
  do 'x.inc';
  report_memory unless $_ % 1e4;
}
---------------------

x.inc contains​:
---------------------
use utf8;
$x = 'x';
---------------------

Perl Info

Flags:
    category=core
    severity=medium

Site configuration information for perl 5.16.3:

Configured by Red Hat, Inc. at Mon Nov 11 12:35:45 UTC 2013.

Summary of my perl5 (revision 5 version 16 subversion 3) configuration:
   
  Platform:
    osname=linux, osvers=3.10.9-200.fc19.x86_64, archname=x86_64-linux-thread-multi
    uname='linux buildvm-01.phx2.fedoraproject.org 3.10.9-200.fc19.x86_64 #1 smp wed aug 21 19:27:58 utc 2013 x86_64 x86_64 x86_64 gnulinux '
    config_args='-des -Doptimize=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -grecord-gcc-switches  -m64 -mtune=generic -Dccdlflags=-Wl,--enable-new-dtags -Dlddlflags=-shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -grecord-gcc-switches  -m64 -mtune=generic -Wl,-z,relro  -DDEBUGGING=-g -Dversion=5.16.3 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dprefix=/usr -Dvendorprefix=/usr -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl5 -Dsitearch=/usr/local/lib64/perl5 -Dprivlib=/usr/share/perl5 -Dvendorlib=/usr/share/perl5/vendor_perl -Darchlib=/usr/lib64/perl5 -Dvendorarch=/usr/lib64/perl5/vendor_perl -Darchname=x86_64-linux-thread-multi -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Duseshrplib -Dusethreads -Duseithreads -Dusedtrace=/usr/bin/dtrace -Duselargefiles -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_sysl!
 og -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dscriptdir=/usr/bin -Dusesitecustomize'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.8.2 20131017 (Red Hat 4.8.2-1)', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -fstack-protector'
    libpth=/usr/local/lib64 /lib64 /usr/lib64
    libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc -lgdbm_compat
    perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.17'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,--enable-new-dtags -Wl,-rpath,/usr/lib64/perl5/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -Wl,-z,relro '

Locally applied patches:
    


@INC for perl 5.16.3:
    /usr/local/lib64/perl5
    /usr/local/share/perl5
    /usr/lib64/perl5/vendor_perl
    /usr/share/perl5/vendor_perl
    /usr/lib64/perl5
    /usr/share/perl5
    .


Environment for perl 5.16.3:
    HOME=/root
    LANG=en_US.utf8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 9, 2014

From @cpansprout

On Fri Feb 07 02​:54​:49 2014, lavv17f@​gmail.com wrote​:

This is a bug report for perl from lavv17f@​gmail.com,
generated with the help of perlbug 1.39 running under perl 5.16.3.

-----------------------------------------------------------------
[Please describe your issue here]
Every time perl includes a file with "use utf8" and text constants
it leaks memory.

Here is my test code​:
---------------------
my $mem = 0;
sub report_memory () {
my $next_mem = qx[ps -p $$ -o size=]+0;
printf "%+8d = %8d K\n", $next_mem-$mem, $next_mem;
$mem = $next_mem;
}

for (0..1e5) {
do 'x.inc';
report_memory unless $_ % 1e4;
}
---------------------

x.inc contains​:
---------------------
use utf8;
$x = 'x';
---------------------

Confirmed. It’s not leaking SVs (which would make the leak
easy to find). I don’t know where it’s leaking.

--

Father Chrysostomos

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 9, 2014

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 9, 2014

From @cpansprout

On Sun Feb 09 07​:04​:24 2014, sprout wrote​:

On Fri Feb 07 02​:54​:49 2014, lavv17f@​gmail.com wrote​:

This is a bug report for perl from lavv17f@​gmail.com,
generated with the help of perlbug 1.39 running under perl 5.16.3.

-----------------------------------------------------------------
[Please describe your issue here]
Every time perl includes a file with "use utf8" and text constants
it leaks memory.

Here is my test code​:
---------------------
my $mem = 0;
sub report_memory () {
my $next_mem = qx[ps -p $$ -o size=]+0;
printf "%+8d = %8d K\n", $next_mem-$mem, $next_mem;
$mem = $next_mem;
}

for (0..1e5) {
do 'x.inc';
report_memory unless $_ % 1e4;
}
---------------------

x.inc contains​:
---------------------
use utf8;
$x = 'x';
---------------------

Confirmed. It’s not leaking SVs (which would make the leak
easy to find). I don’t know where it’s leaking.

perl 5.14.4 in unaffected.

--

Father Chrysostomos

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 9, 2014

From @jkeenan

On Sun Feb 09 07​:05​:42 2014, sprout wrote​:

On Sun Feb 09 07​:04​:24 2014, sprout wrote​:

On Fri Feb 07 02​:54​:49 2014, lavv17f@​gmail.com wrote​:

This is a bug report for perl from lavv17f@​gmail.com,
generated with the help of perlbug 1.39 running under perl 5.16.3.

-----------------------------------------------------------------
[Please describe your issue here]
Every time perl includes a file with "use utf8" and text constants
it leaks memory.

Here is my test code​:
---------------------
my $mem = 0;
sub report_memory () {
my $next_mem = qx[ps -p $$ -o size=]+0;
printf "%+8d = %8d K\n", $next_mem-$mem, $next_mem;
$mem = $next_mem;
}

for (0..1e5) {
do 'x.inc';
report_memory unless $_ % 1e4;
}
---------------------

x.inc contains​:
---------------------
use utf8;
$x = 'x';
---------------------

Confirmed. It’s not leaking SVs (which would make the leak
easy to find). I don’t know where it’s leaking.

perl 5.14.4 in unaffected.

But I don't seem to be getting this error on blead​:

#####
[dromedary-001​:perl] 1006 $ ./perl -Ilib -V
Summary of my perl5 (revision 5 version 19 subversion 9) configuration​:
  Commit id​: 769e486
  Platform​:
  osname=linux, osvers=2.6.32-358.el6.x86_64, archname=x86_64-linux
  uname='linux dromedary-001.ams6.corp.booking.com 2.6.32-358.el6.x86_64 #1 smp fri feb 22 00​:31​:26 utc 2013 x86_64 x86_64 x86_64 gnulinux '
  config_args='-des -Dusedevel'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  use64bitint=define, use64bitall=define, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
...
#####
$ ./perl -I./lib ../p5p/121200_leak.pl
  +620 = 620 K
  +0 = 620 K
  +0 = 620 K
  +0 = 620 K
  +0 = 620 K
  +0 = 620 K
  +0 = 620 K
  +0 = 620 K
  +0 = 620 K
  +0 = 620 K
  +0 = 620 K
#####

Somewhat less clean results on Darwin/PPC, substituting 'rss=' for 'size=', at​:
#####
Summary of my perl5 (revision 5 version 19 subversion 9) configuration​:
  Commit id​: 769e486
  Platform​:
#####
/gitwork/perl/perl -I/gitwork/perl/lib 121200_leak.pl
  +1428 = 1428 K
  -4 = 1424 K
  +20 = 1444 K
  +0 = 1444 K
  +0 = 1444 K
  +0 = 1444 K
  -20 = 1424 K
  +20 = 1444 K
  +0 = 1444 K
  +0 = 1444 K
  +0 = 1444 K
#####

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 10, 2014

From perl5-porters@perl.org

James Keenan wrote​:

But I don't seem to be getting this error on blead​:

I can still reproduce it on blead. It was caused by this commit​:

commit e8b3448
Author​: Brian Fraser <fraserbn@​gmail.com>
Date​: Sat Jun 11 15​:12​:44 2011 -0300

  Cleanup of pad fetching and storing....

Presumably the return value of bytes_from_utf8 needs to be freed.

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 15, 2014

From @Hugmeir

On Mon, Feb 10, 2014 at 1​:16 AM, Father Chrysostomos <sprout@​cpan.org>wrote​:

James Keenan wrote​:

But I don't seem to be getting this error on blead​:

I can still reproduce it on blead. It was caused by this commit​:

commit e8b3448
Author​: Brian Fraser <fraserbn@​gmail.com>
Date​: Sat Jun 11 15​:12​:44 2011 -0300

Cleanup of pad fetching and storing\.\.\.\.

Presumably the return value of bytes_from_utf8 needs to be freed.

Thanks for tracking this one down, Father C! I've been trying to fix this
for a couple of days but kept making ridiculous mistakes. Jetlag's a
wonderful thing :)

My main problem right now is how to add a test for this. Would there be any
objections to stuff like this?

  my $null = File​::Spec->devnull;
  my $ps = IPC​::Cmd​::can_run("ps") && qx[ps -p $$ -o rss= 2>$null]+0
  ? sub { qx[ps -p $$ -o rss= 2>$null]+0 }
  : undef;
  skip("Can't run 'ps' here, skipping", 2) unless $ps;

That makes the test run on my darwin box, but just skips it on my phone,
which has only a barebones version of ps that doesn't understand '-p'.

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 10, 2015

From @unobe

valgrind reports a small memory leak when 'use utf8;' is used in perl 5.20.0, the latest patch release candidate for perl 5.20 (5.20.2-RC1), and the latest development release (5.21.8). The leak is greater when using all optional features (-E). The attached file, leak.pl, can be used to test different versions of perl. I used something similar to leak.pl on blead at the end of last week and no leak was reported.

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 10, 2015

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 10, 2015

From @tonycoz

On Tue Feb 10 10​:04​:58 2015, tiro wrote​:

valgrind reports a small memory leak when 'use utf8;' is used in perl
5.20.0, the latest patch release candidate for perl 5.20 (5.20.2-RC1),
and the latest development release (5.21.8). The leak is greater when
using all optional features (-E). The attached file, leak.pl, can be
used to test different versions of perl. I used something similar to
leak.pl on blead at the end of last week and no leak was reported.

By default perl won't release memory on exit. This is to speed up tear-down of the perl process, since the operating system is going to release that memory anyway when the process exits.

There's a few ways to force the release​:

- if you have a debugging perl, set PERL_DESTRUCT_LEVEL=2

  $ PERL_DESTRUCT_LEVEL=2 valgrind -q --leak-check=yes ~/perl/5.20.1-dbg/bin/perl -e 'use strict'
  $

- creating a thread​:

  $ valgrind -q --leak-check=yes ~/perl/5.20.1-dbg/bin/perl -e 'use strict; use threads; threads->create(sub{})->join'
  $

- the Perl​::Destruct​::Level module on CPAN.

Tony

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 10, 2015

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 10, 2015

From @tonycoz

On Tue Feb 10 15​:23​:31 2015, tonyc wrote​:

There's a few ways to force the release​:

- creating a thread​:

$ valgrind -q --leak-check=yes ~/perl/5.20.1-dbg/bin/perl -e 'use
strict; use threads; threads->create(sub{})->join'
$

Just loading threads is enough, I missed an assignment.

Tony

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 11, 2015

From @unobe

Tony,

I attempted to simply reply to your e-mail but didn't see the reply reflected
here in the bug report so I think perhaps some mail header information was
stripped by my mail client that is required for me to reply via e-mail. Sorry
if it's repetitious.

Thank you for looking at this and for the quick reply.

The test script I previously provided had perl hardcoded so probably wasn't
much use; sorry about that. Also, I should have mentioned that simple
constants like '1' didn't trigger the leak for me.

I didn't have a copy of perl with threads support, so built perl v5.20.1 from
the git source. I found that valgrind still shows definite memory loss when
using utf8, even with threads. Am I missing something?

  $ git checkout v5.20.1
  Note​: checking out 'v5.20.1'.

  You are in 'detached HEAD' state. You can look around, make experimental
  changes and commit them, and you can discard any commits you make in this
  state without impacting any branches by performing another checkout.

  If you want to create a new branch to retain commits you create, you may
  do so (now or later) by using -b with the checkout command again. Example​:

  git checkout -b new_branch_name

  HEAD is now at 4f732e9... perldelta - Finalize acknowledgements

  $ ./Configure -Dusedevel -Dusethreads -Doptimize=-O2 -Dloclibpth='/usr/lib64 /usr/local/lib64' -esd
  [...]

  $ make
  [...]

  $ ./perl -v

  This is perl 5, version 20, subversion 1 (v5.20.1) built for x86_64-linux-thread-multi
  [...]

  $ valgrind -q --leak-check=yes ./perl -Ilib -Mthreads -e '$_'

  $ valgrind -q --leak-check=yes ./perl -Ilib -Mthreads -Mutf8 -e '1'

  $ valgrind -q --leak-check=yes ./perl -Ilib -Mthreads -Mutf8 -e 'use strict'
  ==29658== 5 bytes in 1 blocks are definitely lost in loss record 1 of 6
  ==29658== at 0x4C2845D​: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==29658== by 0x48CCE1​: Perl_safesysmalloc (in /home/unobe/perl)
  ==29658== by 0x5198D5​: Perl_bytes_from_utf8 (in /home/uonbe/perl)
  ==29658== by 0x466806​: Perl_pad_findmy_pvn (in /home/unobe/perl)
  ==29658== by 0x4520BB​: Perl_yylex (in /home/unobe/perl)
  ==29658== by 0x46188F​: Perl_yyparse (in /home/unobe/perl)
  ==29658== by 0x43BDDF​: perl_parse (in /home/unobe/perl)
  ==29658== by 0x41BDC8​: main (in /home/unobe/perl)
  ==29658==

  $ valgrind -q --leak-check=yes ./perl -Ilib -Mthreads -Mutf8 -e '$_'
  ==29671== 3 bytes in 1 blocks are definitely lost in loss record 1 of 6
  ==29671== at 0x4C2845D​: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==29671== by 0x48CCE1​: Perl_safesysmalloc (in /home/unobe/perl)
  ==29671== by 0x5198D5​: Perl_bytes_from_utf8 (in /home/unobe/perl)
  ==29671== by 0x466806​: Perl_pad_findmy_pvn (in /home/unobe/perl)
  ==29671== by 0x451F99​: Perl_yylex (in /home/unobe/perl)
  ==29671== by 0x46188F​: Perl_yyparse (in /home/unobe/perl)
  ==29671== by 0x43BDDF​: perl_parse (in /home/unobe/perl)
  ==29671== by 0x41BDC8​: main (in /home/unobe/perl)
  ==29671==

  $ valgrind -q --leak-check=yes ./perl -Ilib -Mthreads -Mutf8 -e '$_ab'
  ==29673== 5 bytes in 1 blocks are definitely lost in loss record 1 of 6
  ==29673== at 0x4C2845D​: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==29673== by 0x48CCE1​: Perl_safesysmalloc (in /home/unobe/perl)
  ==29673== by 0x5198D5​: Perl_bytes_from_utf8 (in /home/unobe/perl)
  ==29673== by 0x466806​: Perl_pad_findmy_pvn (in /home/unobe/perl)
  ==29673== by 0x451F99​: Perl_yylex (in /home/unobe/perl)
  ==29673== by 0x46188F​: Perl_yyparse (in /home/unobe/perl)
  ==29673== by 0x43BDDF​: perl_parse (in /home/unobe/perl)
  ==29673== by 0x41BDC8​: main (in /home/unobe/perl)
  ==29673==

  $ valgrind -q --leak-check=yes ./perl -Ilib -Mthreads -Mutf8 -e 'sub {}'
  ==29674== 5 bytes in 1 blocks are definitely lost in loss record 1 of 6
  ==29674== at 0x4C2845D​: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==29674== by 0x48CCE1​: Perl_safesysmalloc (in /home/unobe/perl)
  ==29674== by 0x5198D5​: Perl_bytes_from_utf8 (in /home/unobe/perl)
  ==29674== by 0x466806​: Perl_pad_findmy_pvn (in /home/unobe/perl)
  ==29674== by 0x4520BB​: Perl_yylex (in /home/unobe/perl)
  ==29674== by 0x46188F​: Perl_yyparse (in /home/unobe/perl)
  ==29674== by 0x43BDDF​: perl_parse (in /home/unobe/perl)
  ==29674== by 0x41BDC8​: main (in /home/unobe/perl)
  ==29674==

  $ valgrind -q --leak-check=yes ./perl -Ilib -Mthreads -Mutf8 -e 'sub a {}'
  ==29675== 3 bytes in 1 blocks are definitely lost in loss record 1 of 8
  ==29675== at 0x4C2845D​: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==29675== by 0x48CCE1​: Perl_safesysmalloc (in /home/unobe/perl)
  ==29675== by 0x5198D5​: Perl_bytes_from_utf8 (in /home/unobe/perl)
  ==29675== by 0x466806​: Perl_pad_findmy_pvn (in /home/unobe/perl)
  ==29675== by 0x455617​: Perl_yylex (in /home/unobe/perl)
  ==29675== by 0x46188F​: Perl_yyparse (in /home/unobe/perl)
  ==29675== by 0x43BDDF​: perl_parse (in /home/unobe/perl)
  ==29675== by 0x41BDC8​: main (in /home/unobe/perl)
  ==29675==
  ==29675== 3 bytes in 1 blocks are definitely lost in loss record 2 of 8
  ==29675== at 0x4C2845D​: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==29675== by 0x48CCE1​: Perl_safesysmalloc (in /home/unobe/perl)
  ==29675== by 0x5198D5​: Perl_bytes_from_utf8 (in /home/unobe/perl)
  ==29675== by 0x466806​: Perl_pad_findmy_pvn (in /home/unobe/perl)
  ==29675== by 0x451F99​: Perl_yylex (in /home/unobe/perl)
  ==29675== by 0x46188F​: Perl_yyparse (in /home/unobe/perl)
  ==29675== by 0x43BDDF​: perl_parse (in /home/unobe/perl)
  ==29675== by 0x41BDC8​: main (in /home/unobe/perl)
  ==29675==
  ==29675== 5 bytes in 1 blocks are definitely lost in loss record 3 of 8
  ==29675== at 0x4C2845D​: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==29675== by 0x48CCE1​: Perl_safesysmalloc (in /home/unobe/perl)
  ==29675== by 0x5198D5​: Perl_bytes_from_utf8 (in /home/unobe/perl)
  ==29675== by 0x466806​: Perl_pad_findmy_pvn (in /home/unobe/perl)
  ==29675== by 0x4520BB​: Perl_yylex (in /home/unobe/perl)
  ==29675== by 0x46188F​: Perl_yyparse (in /home/unobe/perl)
  ==29675== by 0x43BDDF​: perl_parse (in /home/unobe/perl)
  ==29675== by 0x41BDC8​: main (in /home/unobe/perl)
  ==29675==

  $ valgrind -q --leak-check=yes ./perl -Ilib -Mthreads -Mutf8 -e 'sub ab {}'
  ==29680== 4 bytes in 1 blocks are definitely lost in loss record 1 of 8
  ==29680== at 0x4C2845D​: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==29680== by 0x48CCE1​: Perl_safesysmalloc (in /home/unobe/perl)
  ==29680== by 0x5198D5​: Perl_bytes_from_utf8 (in /home/unobe/perl)
  ==29680== by 0x466806​: Perl_pad_findmy_pvn (in /home/unobe/perl)
  ==29680== by 0x455617​: Perl_yylex (in /home/unobe/perl)
  ==29680== by 0x46188F​: Perl_yyparse (in /home/unobe/perl)
  ==29680== by 0x43BDDF​: perl_parse (in /home/unobe/perl)
  ==29680== by 0x41BDC8​: main (in /home/unobe/perl)
  ==29680==
  ==29680== 4 bytes in 1 blocks are definitely lost in loss record 2 of 8
  ==29680== at 0x4C2845D​: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==29680== by 0x48CCE1​: Perl_safesysmalloc (in /home/unobe/perl)
  ==29680== by 0x5198D5​: Perl_bytes_from_utf8 (in /home/unobe/perl)
  ==29680== by 0x466806​: Perl_pad_findmy_pvn (in /home/unobe/perl)
  ==29680== by 0x451F99​: Perl_yylex (in /home/unobe/perl)
  ==29680== by 0x46188F​: Perl_yyparse (in /home/unobe/perl)
  ==29680== by 0x43BDDF​: perl_parse (in /home/unobe/perl)
  ==29680== by 0x41BDC8​: main (in /home/unobe/perl)
  ==29680==
  ==29680== 5 bytes in 1 blocks are definitely lost in loss record 3 of 8
  ==29680== at 0x4C2845D​: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==29680== by 0x48CCE1​: Perl_safesysmalloc (in /home/unobe/perl)
  ==29680== by 0x5198D5​: Perl_bytes_from_utf8 (in /home/unobe/perl)
  ==29680== by 0x466806​: Perl_pad_findmy_pvn (in /home/unobe/perl)
  ==29680== by 0x4520BB​: Perl_yylex (in /home/unobe/perl)
  ==29680== by 0x46188F​: Perl_yyparse (in /home/unobe/perl)
  ==29680== by 0x43BDDF​: perl_parse (in /home/unobe/perl)
  ==29680== by 0x41BDC8​: main (in /home/unobe/perl)
  ==29680==

Tony Cook via RT wrote on Tue, Feb 10, 2015 at 03​:32​:42PM PST​:

On Tue Feb 10 15​:23​:31 2015, tonyc wrote​:

There's a few ways to force the release​:

- creating a thread​:

$ valgrind -q --leak-check=yes ~/perl/5.20.1-dbg/bin/perl -e 'use
strict; use threads; threads->create(sub{})->join'
$

Just loading threads is enough, I missed an assignment.

Tony

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 12, 2015

From @unobe

Tony,

Thank you for looking at this and for the quick reply.

The test script I previously provided had perl hardcoded so probably wasn't
much use; sorry about that. Also, I should have mentioned that simple
constants like '1' didn't trigger the leak for me.

I didn't have a copy of perl with threads support, so built perl v5.20.1 from
the git source. I found that valgrind still shows definite memory loss when
using utf8, even with threads. Am I missing something?

  $ git checkout v5.20.1
  Note​: checking out 'v5.20.1'.

  You are in 'detached HEAD' state. You can look around, make experimental
  changes and commit them, and you can discard any commits you make in this
  state without impacting any branches by performing another checkout.

  If you want to create a new branch to retain commits you create, you may
  do so (now or later) by using -b with the checkout command again. Example​:

  git checkout -b new_branch_name

  HEAD is now at 4f732e9... perldelta - Finalize acknowledgements

  $ ./Configure -Dusedevel -Dusethreads -Doptimize=-O2 -Dloclibpth='/usr/lib64 /usr/local/lib64' -esd
  [...]

  $ make
  [...]

  $ ./perl -v

  This is perl 5, version 20, subversion 1 (v5.20.1) built for x86_64-linux-thread-multi
  [...]

  $ valgrind -q --leak-check=yes ./perl -Ilib -Mthreads -e '$_'

  $ valgrind -q --leak-check=yes ./perl -Ilib -Mthreads -Mutf8 -e '1'

  $ valgrind -q --leak-check=yes ./perl -Ilib -Mthreads -Mutf8 -e 'use strict'
  ==29658== 5 bytes in 1 blocks are definitely lost in loss record 1 of 6
  ==29658== at 0x4C2845D​: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==29658== by 0x48CCE1​: Perl_safesysmalloc (in /home/unobe/perl)
  ==29658== by 0x5198D5​: Perl_bytes_from_utf8 (in /home/uonbe/perl)
  ==29658== by 0x466806​: Perl_pad_findmy_pvn (in /home/unobe/perl)
  ==29658== by 0x4520BB​: Perl_yylex (in /home/unobe/perl)
  ==29658== by 0x46188F​: Perl_yyparse (in /home/unobe/perl)
  ==29658== by 0x43BDDF​: perl_parse (in /home/unobe/perl)
  ==29658== by 0x41BDC8​: main (in /home/unobe/perl)
  ==29658==

  $ valgrind -q --leak-check=yes ./perl -Ilib -Mthreads -Mutf8 -e '$_'
  ==29671== 3 bytes in 1 blocks are definitely lost in loss record 1 of 6
  ==29671== at 0x4C2845D​: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==29671== by 0x48CCE1​: Perl_safesysmalloc (in /home/unobe/perl)
  ==29671== by 0x5198D5​: Perl_bytes_from_utf8 (in /home/unobe/perl)
  ==29671== by 0x466806​: Perl_pad_findmy_pvn (in /home/unobe/perl)
  ==29671== by 0x451F99​: Perl_yylex (in /home/unobe/perl)
  ==29671== by 0x46188F​: Perl_yyparse (in /home/unobe/perl)
  ==29671== by 0x43BDDF​: perl_parse (in /home/unobe/perl)
  ==29671== by 0x41BDC8​: main (in /home/unobe/perl)
  ==29671==

  $ valgrind -q --leak-check=yes ./perl -Ilib -Mthreads -Mutf8 -e '$_ab'
  ==29673== 5 bytes in 1 blocks are definitely lost in loss record 1 of 6
  ==29673== at 0x4C2845D​: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==29673== by 0x48CCE1​: Perl_safesysmalloc (in /home/unobe/perl)
  ==29673== by 0x5198D5​: Perl_bytes_from_utf8 (in /home/unobe/perl)
  ==29673== by 0x466806​: Perl_pad_findmy_pvn (in /home/unobe/perl)
  ==29673== by 0x451F99​: Perl_yylex (in /home/unobe/perl)
  ==29673== by 0x46188F​: Perl_yyparse (in /home/unobe/perl)
  ==29673== by 0x43BDDF​: perl_parse (in /home/unobe/perl)
  ==29673== by 0x41BDC8​: main (in /home/unobe/perl)
  ==29673==

  $ valgrind -q --leak-check=yes ./perl -Ilib -Mthreads -Mutf8 -e 'sub {}'
  ==29674== 5 bytes in 1 blocks are definitely lost in loss record 1 of 6
  ==29674== at 0x4C2845D​: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==29674== by 0x48CCE1​: Perl_safesysmalloc (in /home/unobe/perl)
  ==29674== by 0x5198D5​: Perl_bytes_from_utf8 (in /home/unobe/perl)
  ==29674== by 0x466806​: Perl_pad_findmy_pvn (in /home/unobe/perl)
  ==29674== by 0x4520BB​: Perl_yylex (in /home/unobe/perl)
  ==29674== by 0x46188F​: Perl_yyparse (in /home/unobe/perl)
  ==29674== by 0x43BDDF​: perl_parse (in /home/unobe/perl)
  ==29674== by 0x41BDC8​: main (in /home/unobe/perl)
  ==29674==

  $ valgrind -q --leak-check=yes ./perl -Ilib -Mthreads -Mutf8 -e 'sub a {}'
  ==29675== 3 bytes in 1 blocks are definitely lost in loss record 1 of 8
  ==29675== at 0x4C2845D​: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==29675== by 0x48CCE1​: Perl_safesysmalloc (in /home/unobe/perl)
  ==29675== by 0x5198D5​: Perl_bytes_from_utf8 (in /home/unobe/perl)
  ==29675== by 0x466806​: Perl_pad_findmy_pvn (in /home/unobe/perl)
  ==29675== by 0x455617​: Perl_yylex (in /home/unobe/perl)
  ==29675== by 0x46188F​: Perl_yyparse (in /home/unobe/perl)
  ==29675== by 0x43BDDF​: perl_parse (in /home/unobe/perl)
  ==29675== by 0x41BDC8​: main (in /home/unobe/perl)
  ==29675==
  ==29675== 3 bytes in 1 blocks are definitely lost in loss record 2 of 8
  ==29675== at 0x4C2845D​: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==29675== by 0x48CCE1​: Perl_safesysmalloc (in /home/unobe/perl)
  ==29675== by 0x5198D5​: Perl_bytes_from_utf8 (in /home/unobe/perl)
  ==29675== by 0x466806​: Perl_pad_findmy_pvn (in /home/unobe/perl)
  ==29675== by 0x451F99​: Perl_yylex (in /home/unobe/perl)
  ==29675== by 0x46188F​: Perl_yyparse (in /home/unobe/perl)
  ==29675== by 0x43BDDF​: perl_parse (in /home/unobe/perl)
  ==29675== by 0x41BDC8​: main (in /home/unobe/perl)
  ==29675==
  ==29675== 5 bytes in 1 blocks are definitely lost in loss record 3 of 8
  ==29675== at 0x4C2845D​: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==29675== by 0x48CCE1​: Perl_safesysmalloc (in /home/unobe/perl)
  ==29675== by 0x5198D5​: Perl_bytes_from_utf8 (in /home/unobe/perl)
  ==29675== by 0x466806​: Perl_pad_findmy_pvn (in /home/unobe/perl)
  ==29675== by 0x4520BB​: Perl_yylex (in /home/unobe/perl)
  ==29675== by 0x46188F​: Perl_yyparse (in /home/unobe/perl)
  ==29675== by 0x43BDDF​: perl_parse (in /home/unobe/perl)
  ==29675== by 0x41BDC8​: main (in /home/unobe/perl)
  ==29675==

  $ valgrind -q --leak-check=yes ./perl -Ilib -Mthreads -Mutf8 -e 'sub ab {}'
  ==29680== 4 bytes in 1 blocks are definitely lost in loss record 1 of 8
  ==29680== at 0x4C2845D​: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==29680== by 0x48CCE1​: Perl_safesysmalloc (in /home/unobe/perl)
  ==29680== by 0x5198D5​: Perl_bytes_from_utf8 (in /home/unobe/perl)
  ==29680== by 0x466806​: Perl_pad_findmy_pvn (in /home/unobe/perl)
  ==29680== by 0x455617​: Perl_yylex (in /home/unobe/perl)
  ==29680== by 0x46188F​: Perl_yyparse (in /home/unobe/perl)
  ==29680== by 0x43BDDF​: perl_parse (in /home/unobe/perl)
  ==29680== by 0x41BDC8​: main (in /home/unobe/perl)
  ==29680==
  ==29680== 4 bytes in 1 blocks are definitely lost in loss record 2 of 8
  ==29680== at 0x4C2845D​: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==29680== by 0x48CCE1​: Perl_safesysmalloc (in /home/unobe/perl)
  ==29680== by 0x5198D5​: Perl_bytes_from_utf8 (in /home/unobe/perl)
  ==29680== by 0x466806​: Perl_pad_findmy_pvn (in /home/unobe/perl)
  ==29680== by 0x451F99​: Perl_yylex (in /home/unobe/perl)
  ==29680== by 0x46188F​: Perl_yyparse (in /home/unobe/perl)
  ==29680== by 0x43BDDF​: perl_parse (in /home/unobe/perl)
  ==29680== by 0x41BDC8​: main (in /home/unobe/perl)
  ==29680==
  ==29680== 5 bytes in 1 blocks are definitely lost in loss record 3 of 8
  ==29680== at 0x4C2845D​: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
  ==29680== by 0x48CCE1​: Perl_safesysmalloc (in /home/unobe/perl)
  ==29680== by 0x5198D5​: Perl_bytes_from_utf8 (in /home/unobe/perl)
  ==29680== by 0x466806​: Perl_pad_findmy_pvn (in /home/unobe/perl)
  ==29680== by 0x4520BB​: Perl_yylex (in /home/unobe/perl)
  ==29680== by 0x46188F​: Perl_yyparse (in /home/unobe/perl)
  ==29680== by 0x43BDDF​: perl_parse (in /home/unobe/perl)
  ==29680== by 0x41BDC8​: main (in /home/unobe/perl)
  ==29680==

Tony Cook via RT wrote on Tue, Feb 10, 2015 at 03​:32​:42PM PST​:

On Tue Feb 10 15​:23​:31 2015, tonyc wrote​:

There's a few ways to force the release​:

- creating a thread​:

$ valgrind -q --leak-check=yes ~/perl/5.20.1-dbg/bin/perl -e 'use
strict; use threads; threads->create(sub{})->join'
$

Just loading threads is enough, I missed an assignment.

Tony

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 25, 2015

From @tonycoz

On Thu Feb 12 08​:06​:08 2015, tiro wrote​:

Tony,

Thank you for looking at this and for the quick reply.

The test script I previously provided had perl hardcoded so probably
wasn't
much use; sorry about that. Also, I should have mentioned that simple
constants like '1' didn't trigger the leak for me.

I didn't have a copy of perl with threads support, so built perl
v5.20.1 from
the git source. I found that valgrind still shows definite memory loss
when
using utf8, even with threads. Am I missing something?

No, it's a real bug, still present in maint-5.20.

The leak isn't from loading utf8.pm itself, but from doing variable name lookups in its context, so with one variable, one block is leaked​:

tony@​mars​:.../git/perl$ valgrind -q --leak-check=full ./perl -Ilib -Mthreads -Mutf8 -e '$x'
==14713== 3 bytes in 1 blocks are definitely lost in loss record 1 of 7
==14713== at 0x4C28BED​: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14713== by 0x4919E1​: Perl_safesysmalloc (in /home/tony/dev/perl/git/perl/perl)
==14713== by 0x522545​: Perl_bytes_from_utf8 (in /home/tony/dev/perl/git/perl/perl)
==14713== by 0x46B4E5​: Perl_pad_findmy_pvn (in /home/tony/dev/perl/git/perl/perl)
==14713== by 0x4562C6​: Perl_yylex (in /home/tony/dev/perl/git/perl/perl)
==14713== by 0x46645F​: Perl_yyparse (in /home/tony/dev/perl/git/perl/perl)
==14713== by 0x43F6A4​: perl_parse (in /home/tony/dev/perl/git/perl/perl)
==14713== by 0x41EDF8​: main (in /home/tony/dev/perl/git/perl/perl)
==14713==

with 3 variables, 3 blocks are leaked​:

tony@​mars​:.../git/perl$ valgrind -q --leak-check=full ./perl -Ilib -Mthreads -Mutf8 -e '$x,$y,$z'
==14737== 9 bytes in 3 blocks are definitely lost in loss record 1 of 7
==14737== at 0x4C28BED​: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==14737== by 0x4919E1​: Perl_safesysmalloc (in /home/tony/dev/perl/git/perl/perl)
==14737== by 0x522545​: Perl_bytes_from_utf8 (in /home/tony/dev/perl/git/perl/perl)
==14737== by 0x46B4E5​: Perl_pad_findmy_pvn (in /home/tony/dev/perl/git/perl/perl)
==14737== by 0x4562C6​: Perl_yylex (in /home/tony/dev/perl/git/perl/perl)
==14737== by 0x46645F​: Perl_yyparse (in /home/tony/dev/perl/git/perl/perl)
==14737== by 0x43F6A4​: perl_parse (in /home/tony/dev/perl/git/perl/perl)
==14737== by 0x41EDF8​: main (in /home/tony/dev/perl/git/perl/perl)
==14737==

This isn't an issue in blead, since the leaking code is no longer present - all pad names are in utf-8 so there's no need to convert.

The attached patch appears to fix the problem for maint-5.20.

Tony

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 25, 2015

From @mauke

Am 25.02.2015 um 01​:22 schrieb Tony Cook via RT​:

The attached patch appears to fix the problem for maint-5.20.

What patch? I see no attachment.

--
Lukas Mai <plokinom@​gmail.com>

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 25, 2015

From @tonycoz

On Wed Feb 25 00​:01​:55 2015, plokinom@​gmail.com wrote​:

Am 25.02.2015 um 01​:22 schrieb Tony Cook via RT​:

The attached patch appears to fix the problem for maint-5.20.

What patch? I see no attachment.

Oops, here it is.

Tony

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 25, 2015

From @tonycoz

0001-perl-123786-don-t-leak-the-temp-utf8-copy-of-namepv.patch
From c701019ebd89baf31d6b03f5200e3f8955bfaa92 Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Wed, 25 Feb 2015 11:20:55 +1100
Subject: [perl #123786] don't leak the temp utf8 copy of namepv

---
 pad.c |    4 ++++
 1 file changed, 4 insertions(+)

diff --git a/pad.c b/pad.c
index fed2892..8d008fd 100644
--- a/pad.c
+++ b/pad.c
@@ -961,6 +961,7 @@ Perl_pad_findmy_pvn(pTHX_ const char *namepv, STRLEN namelen, U32 flags)
     I32 offset;
     const AV *nameav;
     SV **name_svp;
+    bool free_pv = false;
 
     PERL_ARGS_ASSERT_PAD_FINDMY_PVN;
 
@@ -973,6 +974,7 @@ Perl_pad_findmy_pvn(pTHX_ const char *namepv, STRLEN namelen, U32 flags)
     if (flags & padadd_UTF8_NAME) {
         bool is_utf8 = TRUE;
         namepv = (const char*)bytes_from_utf8((U8*)namepv, &namelen, &is_utf8);
+        free_pv = true;
 
         if (is_utf8)
             flags |= padadd_UTF8_NAME;
@@ -982,6 +984,8 @@ Perl_pad_findmy_pvn(pTHX_ const char *namepv, STRLEN namelen, U32 flags)
 
     offset = pad_findlex(namepv, namelen, flags,
                 PL_compcv, PL_cop_seqmax, 1, NULL, &out_sv, &out_flags);
+    if (free_pv)
+        Safefree(namepv);
     if ((PADOFFSET)offset != NOT_IN_PAD) 
 	return offset;
 
-- 
1.7.10.4

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 25, 2015

From @jkeenan

On Tue Feb 24 16​:22​:20 2015, tonyc wrote​:

No, it's a real bug, still present in maint-5.20.

The leak isn't from loading utf8.pm itself, but from doing variable
name lookups in its context, so with one variable, one block is
leaked​:
[snip]

This isn't an issue in blead, since the leaking code is no longer
present - all pad names are in utf-8 so there's no need to convert.

The attached patch appears to fix the problem for maint-5.20.

Tony​:

I checked out tag v5.20.1 and confirmed the leaks reported with valgrind. I then applied your patch, rebuilt (sh ./Configure -des -Dusedevel -Dusethreads -Doptimize=-O2 on Ubuntu 14.04) and confirmed that the leaks had been eliminated.

However, when I checked out blead (commit e11fa37), built with threads as above, and then ran the valgrind tests, memory leaks were still being reported. 2 examples​:

#####
$ valgrind -q --leak-check=yes ./perl -Ilib -Mthreads -e '$_'
==16479== 3,584 (512 direct, 3,072 indirect) bytes in 1 blocks are definitely lost in loss record 18 of 37
==16479== at 0x4C2CC70​: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==16479== by 0x41F32E​: Perl_Slab_Alloc (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x425306​: Perl_newSVOP (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x41C3CE​: S_force_word (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x463D83​: Perl_yylex (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x4696DF​: Perl_yyparse (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x4F2591​: S_doeval (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x4FC8D1​: Perl_pp_require (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x4B5345​: Perl_runops_standard (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x43B54F​: Perl_call_sv (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x43D84E​: Perl_call_list (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x41E286​: S_process_special_blocks (in /home/jkeenan/gitwork/perl/perl)
==16479==
==16479== 23,552 (1,024 direct, 22,528 indirect) bytes in 2 blocks are definitely lost in loss record 35 of 37
==16479== at 0x4C2CC70​: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==16479== by 0x41F32E​: Perl_Slab_Alloc (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x424100​: Perl_newOP (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x46A4A5​: Perl_yyparse (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x4F2591​: S_doeval (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x4FC8D1​: Perl_pp_require (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x4B5345​: Perl_runops_standard (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x43B54F​: Perl_call_sv (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x43D84E​: Perl_call_list (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x41E286​: S_process_special_blocks (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x4343B3​: Perl_newATTRSUB_x (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x437E4C​: Perl_utilize (in /home/jkeenan/gitwork/perl/perl)
==16479==
==16479== 72,192 (1,536 direct, 70,656 indirect) bytes in 3 blocks are definitely lost in loss record 36 of 37
==16479== at 0x4C2CC70​: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==16479== by 0x41F32E​: Perl_Slab_Alloc (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x425306​: Perl_newSVOP (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x41C3CE​: S_force_word (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x46479E​: Perl_yylex (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x4696DF​: Perl_yyparse (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x4F2591​: S_doeval (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x4FC8D1​: Perl_pp_require (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x4B5345​: Perl_runops_standard (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x43B54F​: Perl_call_sv (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x43D84E​: Perl_call_list (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x41E286​: S_process_special_blocks (in /home/jkeenan/gitwork/perl/perl)
==16479==
==16479== 98,816 (3,584 direct, 95,232 indirect) bytes in 7 blocks are definitely lost in loss record 37 of 37
==16479== at 0x4C2CC70​: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==16479== by 0x41F32E​: Perl_Slab_Alloc (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x424100​: Perl_newOP (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x454FB7​: Perl_yylex (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x4696DF​: Perl_yyparse (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x4F2591​: S_doeval (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x4FC8D1​: Perl_pp_require (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x4B5345​: Perl_runops_standard (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x43B54F​: Perl_call_sv (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x43D84E​: Perl_call_list (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x41E286​: S_process_special_blocks (in /home/jkeenan/gitwork/perl/perl)
==16479== by 0x4343B3​: Perl_newATTRSUB_x (in /home/jkeenan/gitwork/perl/perl)
==16479==
#####
$ valgrind -q --leak-check=yes ./perl -Ilib -Mthreads -Mutf8 -e '1'
==16482== 3,584 (512 direct, 3,072 indirect) bytes in 1 blocks are definitely lost in loss record 18 of 37
==16482== at 0x4C2CC70​: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==16482== by 0x41F32E​: Perl_Slab_Alloc (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x425306​: Perl_newSVOP (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x41C3CE​: S_force_word (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x463D83​: Perl_yylex (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x4696DF​: Perl_yyparse (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x4F2591​: S_doeval (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x4FC8D1​: Perl_pp_require (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x4B5345​: Perl_runops_standard (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x43B54F​: Perl_call_sv (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x43D84E​: Perl_call_list (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x41E286​: S_process_special_blocks (in /home/jkeenan/gitwork/perl/perl)
==16482==
==16482== 23,552 (1,024 direct, 22,528 indirect) bytes in 2 blocks are definitely lost in loss record 35 of 37
==16482== at 0x4C2CC70​: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==16482== by 0x41F32E​: Perl_Slab_Alloc (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x424100​: Perl_newOP (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x46A4A5​: Perl_yyparse (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x4F2591​: S_doeval (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x4FC8D1​: Perl_pp_require (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x4B5345​: Perl_runops_standard (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x43B54F​: Perl_call_sv (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x43D84E​: Perl_call_list (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x41E286​: S_process_special_blocks (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x4343B3​: Perl_newATTRSUB_x (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x437E4C​: Perl_utilize (in /home/jkeenan/gitwork/perl/perl)
==16482==
==16482== 72,192 (1,536 direct, 70,656 indirect) bytes in 3 blocks are definitely lost in loss record 36 of 37
==16482== at 0x4C2CC70​: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==16482== by 0x41F32E​: Perl_Slab_Alloc (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x425306​: Perl_newSVOP (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x41C3CE​: S_force_word (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x46479E​: Perl_yylex (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x4696DF​: Perl_yyparse (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x4F2591​: S_doeval (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x4FC8D1​: Perl_pp_require (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x4B5345​: Perl_runops_standard (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x43B54F​: Perl_call_sv (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x43D84E​: Perl_call_list (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x41E286​: S_process_special_blocks (in /home/jkeenan/gitwork/perl/perl)
==16482==
==16482== 98,816 (3,584 direct, 95,232 indirect) bytes in 7 blocks are definitely lost in loss record 37 of 37
==16482== at 0x4C2CC70​: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==16482== by 0x41F32E​: Perl_Slab_Alloc (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x424100​: Perl_newOP (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x454FB7​: Perl_yylex (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x4696DF​: Perl_yyparse (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x4F2591​: S_doeval (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x4FC8D1​: Perl_pp_require (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x4B5345​: Perl_runops_standard (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x43B54F​: Perl_call_sv (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x43D84E​: Perl_call_list (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x41E286​: S_process_special_blocks (in /home/jkeenan/gitwork/perl/perl)
==16482== by 0x4343B3​: Perl_newATTRSUB_x (in /home/jkeenan/gitwork/perl/perl)
==16482==
#####

Based on your remarks, I would have expected blead to be leak-free in this respect. Am I overlooking something? (New to valgrind.)

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Feb 25, 2015

From @tonycoz

On Wed, Feb 25, 2015 at 03​:48​:38PM -0800, James E Keenan via RT wrote​:

However, when I checked out blead (commit e11fa37), built with threads as above, and then ran the valgrind tests, memory leaks were still being reported. 2 examples​:

...

Based on your remarks, I would have expected blead to be leak-free in this respect. Am I overlooking something? (New to valgrind.)

That's a different leak, introduced with the OP_MULTIDEREF
implementation, see https://rt-archive.perl.org/perl5/Ticket/Display.html?id=123922

blead no longer includes the conversion code that produces the leak in
#123786, so the namepv leak isn't in blead.

Tony

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Apr 20, 2015

From @tonycoz

On Mon, Apr 20, 2015 at 02​:13​:23PM +0200, Dirk Stöcker wrote​:

Hello,

I tried to use your patch from
https://rt.perl.org/Public/Bug/Display.html?id=123786
for Perl, as I have servers, which suffer from this a lot (long running tasks :-)

But when the patch is applied, then building perl fails in test stage.

See here for build log​:
https://build.opensuse.org/build/home:dstoecker/openSUSE_13.1/x86_64/perl/_log

Looks much like a double free to me.

Discuss this on the p5p mailing list please.

If you include​:

  [perl #123786]

in the subject it will be included in the ticket.

Tony

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Apr 21, 2015

From @tonycoz

On Mon Apr 20 06​:16​:04 2015, tonyc wrote​:

On Mon, Apr 20, 2015 at 02​:13​:23PM +0200, Dirk Stöcker wrote​:

But when the patch is applied, then building perl fails in test
stage.

The attached patch should fix it.

The interface of bytes_from_utf8() seems to be designed to leak :(

Tony

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Apr 21, 2015

From @tonycoz

0001-perl-123786-don-t-leak-the-temp-utf8-copy-of-namepv.patch
From 9b3706b6b71c09adc56e84dd4e67666e505a33b7 Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Tue, 21 Apr 2015 16:13:13 +1000
Subject: [PATCH] [perl #123786] don't leak the temp utf8 copy of namepv

---
 pad.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/pad.c b/pad.c
index fed2892..f22c3c5 100644
--- a/pad.c
+++ b/pad.c
@@ -976,8 +976,10 @@ Perl_pad_findmy_pvn(pTHX_ const char *namepv, STRLEN namelen, U32 flags)
 
         if (is_utf8)
             flags |= padadd_UTF8_NAME;
-        else
+        else {
             flags &= ~padadd_UTF8_NAME;
+            SAVEFREEPV(namepv);
+        }
     }
 
     offset = pad_findlex(namepv, namelen, flags,
-- 
1.7.10.4

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Jun 12, 2015

From dixoncx@gmail.com

Hi,

This leak is present in perl 5.16 as well.
Is there any patch released for perl 5.16 ?

--
Regards,
Dixon Xavier

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Jun 22, 2015

From @tonycoz

On Fri Jun 12 07​:01​:20 2015, dixoncx@​gmail.com wrote​:

Hi,

This leak is present in perl 5.16 as well.
Is there any patch released for perl 5.16 ?

5.16 is no longer supported by perl5-porters.

From a quick look at pad.c the patch should apply against maint-5.16.

Tony

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Aug 19, 2015

From @steve-m-hay

On Mon Apr 20 23​:19​:30 2015, tonyc wrote​:

On Mon Apr 20 06​:16​:04 2015, tonyc wrote​:

On Mon, Apr 20, 2015 at 02​:13​:23PM +0200, Dirk Stöcker wrote​:

But when the patch is applied, then building perl fails in test
stage.

The attached patch should fix it.

The interface of bytes_from_utf8() seems to be designed to leak :(

Thanks. This fix has now been applied to the maint-5.20 branch in commit 8d89c05 and will shortly be released in perl-5.20.3.

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Aug 19, 2015

@steve-m-hay - Status changed from 'open' to 'pending release'

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Aug 26, 2015

From @ppisar

On 2015-08-19, Steve Hay via RT <perlbug-followup@​perl.org> wrote​:

Thanks. This fix has now been applied to the maint-5.20 branch in
commit 8d89c05 and will shortly be
released in perl-5.20.3.

Is this bug duplicate of bug #121200 (memory leak when including a file with
"use utf8")?

-- Petr

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Aug 27, 2015

From @tonycoz

On Wed Aug 26 04​:57​:08 2015, ppisar wrote​:

On 2015-08-19, Steve Hay via RT <perlbug-followup@​perl.org> wrote​:

Thanks. This fix has now been applied to the maint-5.20 branch in
commit 8d89c05 and will shortly be
released in perl-5.20.3.

Is this bug duplicate of bug #121200 (memory leak when including a file with
"use utf8")?

Certainly looks like it, the test code in 121200 no longer leaks in maint-5.20.

I'll merge the tickets.

Tony

@p5pRT
Copy link
Author

@p5pRT p5pRT commented Aug 27, 2015

@tonycoz - Status changed from 'open' to 'pending release'

@p5pRT
Copy link
Author

@p5pRT p5pRT commented May 13, 2016

From @khwilliamson

Thank you for submitting this report. You have helped make Perl better.
 
With the release of Perl 5.24.0 on May 9, 2016, this and 149 other issues have been resolved.

Perl 5.24.0 may be downloaded via https://metacpan.org/release/RJBS/perl-5.24.0

@p5pRT
Copy link
Author

@p5pRT p5pRT commented May 13, 2016

@khwilliamson - Status changed from 'pending release' to 'resolved'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant