DBI memory leak

[p5pRT]

Migrated from rt.perl.org#56908 (status was 'resolved')

Searchable as RT56908$

[p5pRT]

From lav@yar.ru

Created by lav@yar.ru

Under perl-5.10.0 the following simple program leaks memory,
while under perl-5.8.8 it does not. DBI module version is the
same under both perl versions (1.605). The leak seems to be not
specific to DBD driver, as it also happens with mysql and
oracle drivers.

use DBI;
my $dbh=DBI->connect('DBI​:NullP​:');
while(1) {
  my $q=$dbh->prepare(q{select 1 from dual});
}

Perl Info

Flags:
    category=core
    severity=medium

Site configuration information for perl 5.10.0:

Configured by lav at Thu May 29 18:31:21 MSD 2008.

Summary of my perl5 (revision 5 version 10 subversion 0) configuration:
  Platform:
    osname=solaris, osvers=2.8, archname=sun4-solaris-thread-multi
    uname='sunos night 5.8 generic_117350-39 sun4u sparc '
    config_args='-Dprefix=/usr/local/perl-5.10.0 -Doptimize=-O2 -Dusethreads -des'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-D_REENTRANT -fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='4.3.0', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib '
    libpth=/usr/local/lib /usr/lib /usr/ccs/lib
    libs=-lsocket -lnsl -lgdbm -ldl -lm -lpthread -lc
    perllibs=-lsocket -lnsl -ldl -lm -lpthread -lc
    libc=/lib/libc.so, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' '
    cccdlflags='-fPIC', lddlflags='-G -L/usr/local/lib'

Locally applied patches:
    


@INC for perl 5.10.0:
    /usr/local/perl-5.10.0/lib/5.10.0/sun4-solaris-thread-multi
    /usr/local/perl-5.10.0/lib/5.10.0
    /usr/local/perl-5.10.0/lib/site_perl/5.10.0/sun4-solaris-thread-multi
    /usr/local/perl-5.10.0/lib/site_perl/5.10.0
    .


Environment for perl 5.10.0:
    HOME=/root
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH=/ORACLE/app/oracle/product/9.2.0.1/lib:/usr/lib:/usr/openwin/lib:/usr/openwin/server/lib:/usr/dt/lib:/usr/local/lib
    LOGDIR (unset)
    PATH=/ORACLE/app/oracle/product/9.2.0.1/bin:/usr/local/bin:/opt/bin:/bin:/sbin:/usr/sbin:/usr/ccs/bin:/usr/dt/bin:/usr/openwin/bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

[p5pRT]

From dland@landgren.net

On Mon Jul 14 00​:40​:45 2008, lav@​yar.ru wrote​:

This is a bug report for perl from lav@​yar.ru,
generated with the help of perlbug 1.36 running under perl 5.10.0.

-----------------------------------------------------------------
[Please enter your report here]
Under perl-5.10.0 the following simple program leaks memory,
while under perl-5.8.8 it does not. DBI module version is the
same under both perl versions (1.605). The leak seems to be not
specific to DBD driver, as it also happens with mysql and
oracle drivers.

use DBI;
my $dbh=DBI->connect('DBI​:NullP​:');
while(1) {
my $q=$dbh->prepare(q{select 1 from dual});
}

DBI does some sneaky things with the internals, in the pursuit of raw
performance.

The internals were heavily reworked during the development track up to
5.10, and in the process DBI was dinged a couple of times by changes in
assumptions. As such, and given the absence of response so far (in spite
of a delightfully short test snippet -- thank you), I recommend you post
your report to the dbi-users mailing list.

I've cc'ed Tim Bunce on this reply. It just may turn out that Tim has a
easy solution regardless, and was not aware of the bug report.

Thanks,
David Landgren

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @timbunce

Umm. I've reproduced it using 5.10 but not 5.8.6.

[p5pRT]

From twhaples@airwave.com

I have reproduced this bug with DBI versions 1.604 and 1.607 with Perl
5.10 on Linux (using the DBD​::Pg driver, as well). It looks to be
approximately 72 bytes leaked every time we call ->prepare (specifically
from somewhere in DBI​::_new_handle). Unfortunately our application has
some long-running processes which prepare a lot of database statements.

perl -V information​:

---

Platform​:
  osname=linux, osvers=2.6.9-42.0.3.elsmp,
archname=i386-linux-thread-multi
  uname='linux cent4-rpmbuild.dev.airwave.com 2.6.9-42.0.3.elsmp #1
smp fri oct 6 06​:21​:39 cdt 2006 i686 i686 i386 gnulinux '
  config_args='-des -Doptimize=-O2 -g -march=i386 -mcpu=i686
-Dversion=5.10.0 -Dmyhostname=localhost -Dperladmin=root@​localhost
-Dcc=gcc -Dcf_by=Red Hat, Inc. -Dinstallprefix=/opt/airwave
-Dprefix=/opt/airwave -Dprivlib=/opt/airwave/lib/perl5/5.10.0
-Dsitelib=/opt/airwave/local/lib/perl5/site_perl/5.10.0
-Dvendorlib=/opt/airwave/lib/perl5/vendor_perl/5.10.0
-Darchlib=/opt/airwave/lib/perl5/5.10.0/i386-linux-thread-multi
-Dsitearch=/opt/airwave/local/lib/perl5/site_perl/5.10.0/i386-linux-thread-multi
-Dvendorarch=/opt/airwave/lib/perl5/vendor_perl/5.10.0/i386-linux-thread-multi
-Darchname=i386-linux-thread-multi -Dvendorprefix=/opt/airwave
-Dsiteprefix=/opt/airwave/local -Duseshrplib -Dusethreads -Duseithreads
-Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm
-Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n
-Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr
-Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto
-Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto
-Ud_setservent_r_proto -Dscriptdir=/opt/airwave/bin'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=define, usemultiplicity=define
  useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
  use64bitint=undef, use64bitall=undef, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBUGGING
-fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
  optimize='-O2 -g -march=i386 -mcpu=i686',
  cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBUGGING
-fno-strict-aliasing -pipe -I/usr/local/include -I/usr/include/gdbm'
  ccversion='', gccversion='3.4.6 20060404 (Red Hat 3.4.6-9)',
gccosandvers=''
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
  ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
  alignbytes=4, prototype=define
  Linker and Libraries​:
  ld='gcc', ldflags =' -L/usr/local/lib'
  libpth=/usr/local/lib /lib /usr/lib
  libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
  perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
  libc=/lib/libc-2.3.4.so, so=so, useshrplib=true, libperl=libperl.so
  gnulibc_version='2.3.4'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E
-Wl,-rpath,/opt/airwave/lib/perl5/5.10.0/i386-linux-thread-multi/CORE'
  cccdlflags='-fPIC', lddlflags='-shared -O2 -g -march=i386 -mcpu=i686
-L/usr/local/lib'

Characteristics of this binary (from libperl)​:
  Compile-time options​: DEBUGGING MULTIPLICITY PERL_DONT_CREATE_GVSV
  PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP
  PERL_TRACK_MEMPOOL USE_ITHREADS USE_LARGE_FILES
  USE_PERLIO USE_REENTRANT_API
  Built under linux
  Compiled at Jun 30 2008 15​:26​:47

[p5pRT]

From p5p@perl.wizbit.be

Citeren "lav@​yar.ru (via RT)" <perlbug-followup@​perl.org>​:

# New Ticket Created by lav@​yar.ru
# Please include the string​: [perl #56908]
# in the subject line of all future correspondence about this issue.
# <URL​: http​://rt.perl.org/rt3/Ticket/Display.html?id=56908 >

This is a bug report for perl from lav@​yar.ru,
generated with the help of perlbug 1.36 running under perl 5.10.0.

-----------------------------------------------------------------
[Please enter your report here]
Under perl-5.10.0 the following simple program leaks memory,
while under perl-5.8.8 it does not. DBI module version is the
same under both perl versions (1.605). The leak seems to be not
specific to DBD driver, as it also happens with mysql and
oracle drivers.

use DBI;
my $dbh=DBI->connect('DBI​:NullP​:');
while(1) {
my $q=$dbh->prepare(q{select 1 from dual});
}

I've done a binary search on it with these results​: (with DBI-1.607)

----Program----
#!/usr/bin/perl -l

use DBI;

my $dbh=DBI->connect('DBI​:NullP​:');
my $i = 0;
my $mem1 = qx/ps u $$/;

while ($i++ < 100_000) {
  my $q=$dbh->prepare(q{select 1 from dual});
}

my $mem2 = qx/ps u $$/;

my ($vsz1) = $mem1 =~ m/^perl\s+\d+\s+[0-9.]+\s+[0-9.]+\s+(\d+)/m;
my ($vsz2) = $mem2 =~ m/^perl\s+\d+\s+[0-9.]+\s+[0-9.]+\s+(\d+)/m;
if ($vsz1 + 1000 < $vsz2) {
  print "not ok";
}
else {
  print "ok";
}

----Output of .../pUmCekJ/perl-5.9.2@​26529/bin/perl----
ok

----EOF ($?='0')----
----Output of .../pn6SMVj/perl-5.9.2@​26530/bin/perl----
not ok

----EOF ($?='0')----

The output with blead is the same. (meaning​: it hasn't changed since)

http​://public.activestate.com/cgi-bin/perlbrowse/p/26530
Change 26530 by nicholas@​nicholas-saigo on 2005/12/30 01​:08​:46

  RMAGIC on symbol tables is bad, m'kay.
  Allow hashes (and therefore all symbol tables) to store the
  backreference array in the hv_aux structure, and thereby undo the
  performance damage of 24966, which resulted in 60% of all hash lookups
  trying to mg_find tiehash magic.

I'll leave it up to someone else to figure out if it's a bug in the
CORE or a bug in DBI...

Kind regards,

Bram

[p5pRT]

From @timbunce

On Tue, Aug 05, 2008 at 10​:00​:12PM +0200, Bram wrote​:

Citeren "lav@​yar.ru (via RT)" <perlbug-followup@​perl.org>​:

# <URL​: http​://rt.perl.org/rt3/Ticket/Display.html?id=56908 >

Under perl-5.10.0 the following simple program leaks memory,
while under perl-5.8.8 it does not. DBI module version is the
same under both perl versions (1.605). The leak seems to be not
specific to DBD driver, as it also happens with mysql and
oracle drivers.

use DBI;
my $dbh=DBI->connect('DBI​:NullP​:');
while(1) {
my $q=$dbh->prepare(q{select 1 from dual});
}

I've done a binary search on it with these results​: (with DBI-1.607)

The output with blead is the same. (meaning​: it hasn't changed since)

http​://public.activestate.com/cgi-bin/perlbrowse/p/26530
Change 26530 by nicholas@​nicholas-saigo on 2005/12/30 01​:08​:46

RMAGIC on symbol tables is bad\, m'kay\.
Allow hashes \(and therefore all symbol tables\) to store the
backreference array in the hv\_aux structure\, and thereby undo the
performance damage of 24966\, which resulted in 60% of all hash lookups
trying to mg\_find tiehash magic\.

I'll leave it up to someone else to figure out if it's a bug in the CORE or
a bug in DBI...

Having looked at the patch, and felt my eyes glazing over, I'm going to
point the finger at Nick first :)

Nick, in case it helps... DBI uses weakrefs for only one thing.
Whenever a new handle is created the parent handle's ChildHandles
attribute (an array ref) gets a weak ref to the new child pushed onto it.
This is the code that does that (search for weak in DBI.xs)​:

  AV *av;
  /* add weakref to new (outer) handle into parents ChildHandles array */
  tmp_svp = hv_fetch((HV*)SvRV(parent), "ChildHandles", 12, 1);
  if (!SvROK(*tmp_svp)) {
  SV *ChildHandles_rvav = newRV_noinc((SV*)newAV());
  sv_setsv(*tmp_svp, ChildHandles_rvav);
  sv_free(ChildHandles_rvav);
  }
  av = (AV*)SvRV(*tmp_svp);
  av_push(av, (SV*)sv_rvweaken(newRV((SV*)SvRV(orv))));
  if (av_len(av) % 120 == 0) {
  /* time to do some housekeeping to remove dead handles */
  I32 i = av_len(av); /* 0 = 1 element */
  while (i-- >= 0) {
  SV *sv = av_shift(av);
  if (SvOK(sv))
  av_push(av, sv);
  else
  sv_free(sv);
  }
  }

Let me know if that's doing anything wrong, or sub-optimally.
(I can see a few cleanups but nothing fundamentally wrong.)

One thing that's unusual is that these are weakrefs to *tied* hashes.
Perhaps that's an odd case not handled by the patch.

Tim.

[p5pRT]

From @nwc10

On Wed, Aug 06, 2008 at 12​:12​:44PM +0100, Tim Bunce wrote​:

Nick, in case it helps... DBI uses weakrefs for only one thing.
Whenever a new handle is created the parent handle's ChildHandles
attribute (an array ref) gets a weak ref to the new child pushed onto it.
This is the code that does that (search for weak in DBI.xs)​:

One thing that's unusual is that these are weakrefs to *tied* hashes.
Perhaps that's an odd case not handled by the patch.

Well, it should have still worked...

I doubt that I'll get any time to look at this this side of September.

Nicholas Clark

[p5pRT]

From @iabyn

fixed in bleed by change 34209, also integrated into 5.10.1

[p5pRT]

@iabyn - Status changed from 'open' to 'resolved'