problems with Scalar::Util::weaken() and CLONE under ithreads

[p5pRT]

Migrated from rt.perl.org#24660 (status was 'resolved')

Searchable as RT24660$

[p5pRT]

From stas@stason.org

Scalar​::Util​::weaken() proposed to use in the conjunction with the CLONE
function to solve the problem of cloning non-perl objects suffers itself from
cloning.

Here is an example​:

#/tmp/weaken2
package main;
use threads;
use Scalar​::Util;
my $data = "a";
my $obj = \$data;
my $copy = $obj;
Scalar​::Util​::weaken($copy);
threads->new(sub { print "thread started\n";})->detach for (1..1);
sleep 1;

% perl-5.8.1-ithread /tmp/weaken2
thread started
Attempt to free unreferenced scalar SV 0x8154f74.

Oops, a weakened $copy was cloned too, but its guts are now messed
up. What scalar do you ask is unreferenced? It's the magic that was
added by weaken() and cloned during perl_clone, that perl complains about.
Here is a more complex program that proves that with Devel​::Peek​:

#/tmp/weaken
package Foo;

use strict;
use warnings;

use Scalar​::Util;
use Devel​::Peek;

my %objects;
sub new {
  my $data = "a";
  my $self = bless \$data, __PACKAGE__;
  my $copy = $self;
  Scalar​::Util​::weaken($copy);

  $objects{"$self"} = $copy;
  return $self;
}

sub DESTROY {
  warn "DESTROY\n";
}

sub CLONE {
  warn "CLONE is called";
  for my $key ( keys %objects) {
  my $self = delete $objects{$key};
  Dump $self;
  }
}

package main;
use threads;

my $obj = Foo->new;

threads->new(sub { print "thread started\n";})->detach for (1..1);
sleep 1;

% perl-5.8.1-ithread /tmp/weaken
CLONE is called at /tmp/weaken line 23.
SV = RV(0x8110b2c) at 0x815de1c
  REFCNT = 1
  FLAGS = (PADBUSY,PADMY,ROK)
  RV = 0x8156e1c
  SV = PVMG(0x8115e38) at 0x8156e1c
  REFCNT = 2
  FLAGS = (PADBUSY,PADMY,OBJECT,RMG,POK,pPOK)
  IV = 0
  NV = 0
  PV = 0x8157860 "a"\0
  CUR = 1
  LEN = 2
  MAGIC = 0x8157828
  MG_VIRTUAL = &PL_vtbl_backref
  MG_TYPE = PERL_MAGIC_backref(<)
  MG_FLAGS = 0x02
  REFCOUNTED
  MG_OBJ = 0x8156e28
  SV = PVAV(0x8153494) at 0x8156e28
  REFCNT = 1
  FLAGS = ()
  IV = 0
  NV = 0
  ARRAY = 0x8157848
  FILL = 0
  MAX = 3
  ARYLEN = 0x0
  FLAGS = (REAL)
  Elt No. 0
  STASH = 0x8156d2c "Foo"
thread started
DESTROY
Attempt to free unreferenced scalar SV 0x8156e28 during global destruction.
DESTROY

As you can see SV 0x8156e28 is​:

  SV = PVAV(0x8153494) at 0x8156e28
  REFCNT = 1
  FLAGS = ()
  IV = 0
  NV = 0
  ARRAY = 0x8157848
  FILL = 0
  MAX = 3
  ARYLEN = 0x0
  FLAGS = (REAL)
  Elt No. 0

Any chance we can change perl to call Devel​::Peek​::Dump when producing​:

  Attempt to free unreferenced scalar SV 0x8154f74.

would save us a lot of time, trying to figure out what SV went lost.

As Liz has pointed out​:

The problem with the weakened scalar does not exist in 5.8.0, but does exist
in 5.8.1 and 5.8.2. So it must have happened somewhere between 5.8.0 and 5.8.1.

perl-5.8.1-ithread -V
Summary of my perl5 (revision 5.0 version 8 subversion 1) configuration​:
  Platform​:
  osname=linux, osvers=2.4.21-0.18mdkcustom, archname=i686-linux-thread-multi
  uname='linux rabbit.stason.org 2.4.21-0.18mdkcustom #6 mon jun 16
16​:26​:34 est 2003 i686 unknown unknown gnulinux '
  config_args='-des -Dprefix=/home/stas/perl/5.8.1-ithread -Dusethreads
-Doptimize=-g -Duseshrplib -Dusedevel'
  hint=recommended, useposix=true, d_sigaction=define
  usethreads=define use5005threads=undef useithreads=define
usemultiplicity=define
  useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
  use64bitint=undef use64bitall=undef uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS
-DDEBUGGING -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
  optimize='-g',
  cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING
-fno-strict-aliasing -I/usr/local/include -I/usr/include/gdbm'
  ccversion='', gccversion='3.3.1 (Mandrake Linux 9.2 3.3.1-2mdk)',
gccosandvers=''
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
  ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
  alignbytes=4, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -L/usr/local/lib'
  libpth=/usr/local/lib /lib /usr/lib
  libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
  perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
  libc=/lib/libc-2.3.2.so, so=so, useshrplib=true, libperl=libperl.so
  gnulibc_version='2.3.2'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic
-Wl,-rpath,/home/stas/perl/5.8.1-ithread/lib/5.8.1/i686-linux-thread-multi/CORE'
  cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'

Characteristics of this binary (from libperl)​:
  Compile-time options​: DEBUGGING MULTIPLICITY USE_ITHREADS USE_LARGE_FILES
PERL_IMPLICIT_CONTEXT
  Built under linux
  Compiled at Oct 9 2003 11​:57​:40
  %ENV​:
  PERLDOC_PAGER="less -R"
  @​INC​:
  /home/stas/perl/5.8.1-ithread/lib/5.8.1/i686-linux-thread-multi
  /home/stas/perl/5.8.1-ithread/lib/5.8.1
  /home/stas/perl/5.8.1-ithread/lib/site_perl/5.8.1/i686-linux-thread-multi
  /home/stas/perl/5.8.1-ithread/lib/site_perl/5.8.1
  /home/stas/perl/5.8.1-ithread/lib/site_perl
  .

[p5pRT]

From enache@rdslink.ro

The patch below apparently fixes the problems with sv_rvweaken and ithreads.

The idea of "backreferences" is bogus in the first place​: there are
probably still a lot of problems with it.

I'm not sure it's correct​: however it's the only thing I found which
avoids all of loud and silent leaks, wrong reference counts, etc.

Regards,
Adi

Inline Patch

--- /arc/bleadperl/sv.c	2003-12-12 01:22:45.000000000 +0200
+++ ./sv.c	2003-12-18 22:30:18.000000000 +0200
@@ -5137,15 +5137,13 @@ S_sv_add_backref(pTHX_ SV *tsv, SV *sv)
 	 * by magic_killbackrefs() when tsv is being freed */
     }
     if (AvFILLp(av) >= AvMAX(av)) {
+        I32 i;
         SV **svp = AvARRAY(av);
-        I32 i = AvFILLp(av);
-        while (i >= 0) {
-            if (svp[i] == &PL_sv_undef) {
+        for (i = AvFILLp(av); i >= 0; i--)
+            if (!svp[i]) {
                 svp[i] = sv;        /* reuse the slot */
                 return;
             }
-            i--;
-        }
         av_extend(av, AvFILLp(av)+1);
     }
     AvARRAY(av)[++AvFILLp(av)] = sv; /* av_push() */
@@ -5167,13 +5165,8 @@ S_sv_del_backref(pTHX_ SV *sv)
 	Perl_croak(aTHX_ "panic: del_backref");
     av = (AV *)mg->mg_obj;
     svp = AvARRAY(av);
-    i = AvFILLp(av);
-    while (i >= 0) {
-	if (svp[i] == sv) {
-	    svp[i] = &PL_sv_undef; /* XXX */
-	}
-	i--;
-    }
+    for (i = AvFILLp(av); i >= 0; i--)
+	if (svp[i] == sv) svp[i] = Nullsv;
 }
 
 /*
@@ -9793,16 +9786,15 @@ Perl_mg_dup(pTHX_ MAGIC *mg, CLONE_PARAM
 	    nmg->mg_obj	= (SV*)re_dup((REGEXP*)mg->mg_obj, param);
 	}
 	else if(mg->mg_type == PERL_MAGIC_backref) {
-	     AV *av = (AV*) mg->mg_obj;
-	     SV **svp;
-	     I32 i;
-	     nmg->mg_obj = (SV*)newAV();
-	     svp = AvARRAY(av);
-	     i = AvFILLp(av);
-	     while (i >= 0) {
-		  av_push((AV*)nmg->mg_obj,sv_dup(svp[i],param));
-		  i--;
-	     }
+	    AV *av = (AV*) mg->mg_obj;
+	    SV **svp;
+	    I32 i;
+	    SvREFCNT_inc(nmg->mg_obj = (SV*)newAV());
+	    svp = AvARRAY(av);
+	    for (i = AvFILLp(av); i >= 0; i--) {
+		if (!svp[i] || SvREFCNT(svp[i]) < 2) continue;
+		av_push((AV*)nmg->mg_obj,sv_dup(svp[i],param));
+	    }
 	}
 	else {
 	    nmg->mg_obj	= (mg->mg_flags & MGf_REFCOUNTED)
--- /arc/bleadperl/mg.c	2003-12-15 11:33:01.000000000 +0200
+++ ./mg.c	2003-12-18 22:25:44.000000000 +0200
@@ -1927,14 +1927,14 @@ Perl_magic_killbackrefs(pTHX_ SV *sv, MA
     SV **svp = AvARRAY(av);
     I32 i = AvFILLp(av);
     while (i >= 0) {
-	if (svp[i] && svp[i] != &PL_sv_undef) {
+	if (svp[i]) {
 	    if (!SvWEAKREF(svp[i]))
 		Perl_croak(aTHX_ "panic: magic_killbackrefs");
 	    /* XXX Should we check that it hasn't changed? */
 	    SvRV(svp[i]) = 0;
 	    (void)SvOK_off(svp[i]);
 	    SvWEAKREF_off(svp[i]);
-	    svp[i] = &PL_sv_undef;
+	    svp[i] = Nullsv;
 	}
 	i--;
     }

[p5pRT]

From stas@stason.org

Enache Adrian wrote​:

The patch below apparently fixes the problems with sv_rvweaken and ithreads.

The idea of "backreferences" is bogus in the first place​: there are
probably still a lot of problems with it.

I'm not sure it's correct​: however it's the only thing I found which
avoids all of loud and silent leaks, wrong reference counts, etc.

Thanks a lot, Enache!!!

It indeed solves 2 cases in my report #24660 and 1 case in Liz's #24663, but
the problem has not completely gone​: Example​::CLONE 'make test' still has this
problem​:

make test
...
PERL_DL_NONLAZY=1 /home/stas/perl/blead-ithread/bin/perl5.9.0
"-MExtUtils​::Command​::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/basic......ok
t/threads....ok 9/17Attempt to free unreferenced scalar​: SV 0x40179e5c at
/home/stas/work/modules/Example-CLONE/blib/lib/Example/CLONE.pm line 47.
t/threads....ok 13/17Attempt to free unreferenced scalar​: SV 0x40179e5c at
/home/stas/work/modules/Example-CLONE/blib/lib/Example/CLONE.pm line 47.
t/threads....ok
All tests successful.

You can get this test package from here​:
http​://apache.org/~stas/Example-CLONE-0.01.tar.gz

The problem seems to be in lib/Example/CLONE.pm​:

  *CLONE = sub {
  for my $key ( keys %objects) {
  my $self = delete $objects{$key};
  ^^^^^^^^
If I rewrite the above code as​:

  my $self = $objects{$key};
  delete $objects{$key};

the problem disappears.

__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http​://stason.org/ mod_perl Guide ---> http​://perl.apache.org
mailto​:stas@​stason.org http​://use.perl.org http​://apacheweek.com
http​://modperlbook.org http​://apache.org http​://ticketmaster.com

[p5pRT]

From enache@rdslink.ro

On Thu, Dec 18, 2003 a.d., Stas Bekman wrote​:

It indeed solves 2 cases in my report #24660 and 1 case in Liz's #24663,
but the problem has not completely gone​: Example​::CLONE 'make test' still
has this problem​:

make test
...
PERL_DL_NONLAZY=1 /home/stas/perl/blead-ithread/bin/perl5.9.0
"-MExtUtils​::Command​::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')"
t/*.t
t/basic......ok
t/threads....ok 9/17Attempt to free unreferenced scalar​: SV 0x40179e5c at
/home/stas/work/modules/Example-CLONE/blib/lib/Example/CLONE.pm line 47.
t/threads....ok 13/17Attempt to free unreferenced scalar​: SV 0x40179e5c at
/home/stas/work/modules/Example-CLONE/blib/lib/Example/CLONE.pm line 47.
t/threads....ok
All tests successful.

Hmm, I cannot reproduce that.

$ make test
...
PERL_DL_NONLAZY=1 /opt/y/perl/5.9.0/i686-linux-thread-multi-64int-ld/bin/perl "-MExtUtils​::Command​::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/basic......ok
t/threads....ok
All tests successful.
Files=2, Tests=25, 1 wallclock secs ( 0.74 cusr + 0.12 csys = 0.86 CPU)

I'll try with a non-DEBUGGING perl too ...

Regards,
Adi

[p5pRT]

From stas@stason.org

Adrian Enache via RT wrote​:

On Thu, Dec 18, 2003 a.d., Stas Bekman wrote​:

It indeed solves 2 cases in my report #24660 and 1 case in Liz's #24663,
but the problem has not completely gone​: Example​::CLONE 'make test' still
has this problem​:

make test
...
PERL_DL_NONLAZY=1 /home/stas/perl/blead-ithread/bin/perl5.9.0
"-MExtUtils​::Command​::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')"
t/*.t
t/basic......ok
t/threads....ok 9/17Attempt to free unreferenced scalar​: SV 0x40179e5c at
/home/stas/work/modules/Example-CLONE/blib/lib/Example/CLONE.pm line 47.
t/threads....ok 13/17Attempt to free unreferenced scalar​: SV 0x40179e5c at
/home/stas/work/modules/Example-CLONE/blib/lib/Example/CLONE.pm line 47.
t/threads....ok
All tests successful.

Hmm, I cannot reproduce that.

$ make test
...
PERL_DL_NONLAZY=1 /opt/y/perl/5.9.0/i686-linux-thread-multi-64int-ld/bin/perl "-MExtUtils​::Command​::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/basic......ok
t/threads....ok
All tests successful.
Files=2, Tests=25, 1 wallclock secs ( 0.74 cusr + 0.12 csys = 0.86 CPU)

I'll try with a non-DEBUGGING perl too ...

Mine is DEBUGGING

May be this will help​:

My blead is #21917

My perl -V​:

perl-blead-ithread -V
Summary of my perl5 (revision 5.0 version 9 subversion 0 patch 21917)
configuration​:
  Platform​:
  osname=linux, osvers=2.4.22-10mdk, archname=i686-linux-thread-multi
  uname='linux rabbit.stason.org 2.4.22-10mdk #1 thu sep 18 12​:30​:58 cest
2003 i686 unknown unknown gnulinux '
  config_args='-des -Dprefix=/home/stas/perl/blead-ithread -Dusethreads
-Doptimize=-g -Duseshrplib -Dusedevel'
  hint=recommended, useposix=true, d_sigaction=define
  usethreads=define useithreads=define usemultiplicity=define
  useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
  use64bitint=undef use64bitall=undef uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS
-DDEBUGGING -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
  optimize='-g',
  cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING
-fno-strict-aliasing -I/usr/local/include -I/usr/include/gdbm'
  ccversion='', gccversion='3.3.1 (Mandrake Linux 9.2 3.3.1-4mdk)',
gccosandvers=''
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
  ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
  alignbytes=4, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -L/usr/local/lib'
  libpth=/usr/local/lib /lib /usr/lib
  libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
  perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
  libc=/lib/libc-2.3.2.so, so=so, useshrplib=true, libperl=libperl.so
  gnulibc_version='2.3.2'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic
-Wl,-rpath,/home/stas/perl/blead-ithread/lib/5.9.0/i686-linux-thread-multi/CORE'
  cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'

Characteristics of this binary (from libperl)​:
  Compile-time options​: DEBUGGING MULTIPLICITY USE_ITHREADS USE_LARGE_FILES
PERL_IMPLICIT_CONTEXT
  Locally applied patches​:
  DEVEL21538
  Built under linux
  Compiled at Dec 16 2003 11​:20​:18
  %ENV​:
  PERLDOC_PAGER="less -R"
  @​INC​:
  /home/stas/perl/blead-ithread/lib/5.9.0/i686-linux-thread-multi
  /home/stas/perl/blead-ithread/lib/5.9.0
  /home/stas/perl/blead-ithread/lib/site_perl/5.9.0/i686-linux-thread-multi
  /home/stas/perl/blead-ithread/lib/site_perl/5.9.0
  /home/stas/perl/blead-ithread/lib/site_perl
  .

__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http​://stason.org/ mod_perl Guide ---> http​://perl.apache.org
mailto​:stas@​stason.org http​://use.perl.org http​://apacheweek.com
http​://modperlbook.org http​://apache.org http​://ticketmaster.com

[p5pRT]

From enache@rdslink.ro

On Thu, Dec 18, 2003 a.d., Stas Bekman wrote​:

osname=linux\, osvers=2\.4\.22\-10mdk\, archname=i686\-linux\-thread\-multi

...

config\_args='\-des \-Dprefix=/home/stas/perl/blead\-ithread \-Dusethreads 

-Doptimize=-g -Duseshrplib -Dusedevel'
...
libc=/lib/libc-2.3.2.so, so=so, useshrplib=true, libperl=libperl.so
...
gnulibc_version='2.3.2'

I suspect the perl you're using to build Example-CLONE is picking
the old shared library.

Without the patch, the tests barf as you describe; with the patch they
run just fine here (linux, shared libperl.so, etc - very similar to yours)

Regards,
Adi

[p5pRT]

From stas@stason.org

Enache Adrian wrote​:

On Thu, Dec 18, 2003 a.d., Stas Bekman wrote​:

osname=linux, osvers=2.4.22-10mdk, archname=i686-linux-thread-multi

...

config_args='-des -Dprefix=/home/stas/perl/blead-ithread -Dusethreads
-Doptimize=-g -Duseshrplib -Dusedevel'

...

libc=/lib/libc-2.3.2.so, so=so, useshrplib=true, libperl=libperl.so

...

gnulibc_version='2.3.2'

I suspect the perl you're using to build Example-CLONE is picking
the old shared library.

It can't pick the old library since it's been overwritten by the new one, I
did 'make install' and checked that it's new. Besides this exact perl has no
problems with 3 other tests reported earlier, but it does with Example​::CLONE
test.

Without the patch, the tests barf as you describe; with the patch they
run just fine here (linux, shared libperl.so, etc - very similar to yours)

But I know what's going on. I get these failures at random. So your must
repeate several times to get them. Or may need to put your machine under some
load. We are talking about t/threads.t The failure of test depends on the
timing of the spawned threads and the parent thread's exit times. Try to play
with sleep()/select in read_test and at the main thread​:

For example if I add 'sleep 1' to read_test, half of the test fails
completely, because something is broken in threads. If I make the parent
thread sleep long enough till all spawned threads are done I get all sub-tests
pass and no 'Attempt to free unreferenced scalar' messages. But this is
obviously wrong.

...
threads->new(\&read_test)->detach for 1..$threads;
sub read_test : locked {
  for my $count (1..2) {
  my $expected_count = $global_count+$count;
  my $received = $obj->read;
  is $received, $expected_str, "storage verification";
  is $obj->count, $expected_count, "read $expected_count times";
  sleep 1;
  ^^^^^^^^
  }
}

# workaround the situations where the main thread exits before the
# child threads, which shouldn't be a problem since all threads are
# detached, but something is broken in perl
select(undef, undef, undef, 5.25);
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I hope you will be able to reproduce it.

BTW, I have just a single processor.

__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http​://stason.org/ mod_perl Guide ---> http​://perl.apache.org
mailto​:stas@​stason.org http​://use.perl.org http​://apacheweek.com
http​://modperlbook.org http​://apache.org http​://ticketmaster.com

[p5pRT]

From enache@rdslink.ro

On Thu, Dec 18, 2003 a.d., Stas Bekman wrote​:

But I know what's going on. I get these failures at random. So your must
repeate several times to get them. Or may need to put your machine under
some load. We are talking about t/threads.t The failure of test depends on
the timing of the spawned threads and the parent thread's exit times. Try
to play with sleep()/select in read_test and at the main thread​:

For example if I add 'sleep 1' to read_test, half of the test fails
completely, because something is broken in threads. If I make the parent
thread sleep long enough till all spawned threads are done I get all
sub-tests pass and no 'Attempt to free unreferenced scalar' messages. But
this is obviously wrong.

That's right. But that problem doesn't seem to be related to your use
of Scalar​::Util​::weaken().
For now I'll check if it's something with CLONE ...

Regards,
Adi

[p5pRT]

From stas@stason.org

Adrian Enache via RT wrote​:

On Thu, Dec 18, 2003 a.d., Stas Bekman wrote​:

But I know what's going on. I get these failures at random. So your must
repeate several times to get them. Or may need to put your machine under
some load. We are talking about t/threads.t The failure of test depends on
the timing of the spawned threads and the parent thread's exit times. Try
to play with sleep()/select in read_test and at the main thread​:

For example if I add 'sleep 1' to read_test, half of the test fails
completely, because something is broken in threads. If I make the parent
thread sleep long enough till all spawned threads are done I get all
sub-tests pass and no 'Attempt to free unreferenced scalar' messages. But
this is obviously wrong.

That's right.

That means that you were able to reproduce it, right?

But that problem doesn't seem to be related to your use
of Scalar​::Util​::weaken().

I thought it does, since it happens when you delete a weakened var. But I
could be wrong.

For now I'll check if it's something with CLONE .

I'm seeing a similar error when deleting a totally different thing from a hash
(code ref) and it's w/o threads. I'm trying to reproduce it and see if it's
related.

__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http​://stason.org/ mod_perl Guide ---> http​://perl.apache.org
mailto​:stas@​stason.org http​://use.perl.org http​://apacheweek.com
http​://modperlbook.org http​://apache.org http​://ticketmaster.com

[p5pRT]

From enache@rdslink.ro

On Fri, Dec 19, 2003 a.d., Stas Bekman wrote​:

That means that you were able to reproduce it, right?

Unfortunately, yes :-)

For now I'll check if it's something with CLONE .

It looks worse​:

$ perl -mthreads -e 'threads->new(sub{my%h=(1,2);delete $h{1}})->join for 1,2'
Attempt to free unreferenced scalar​: SV 0x4017fe84 at -e line 1.

I'm seeing a similar error when deleting a totally different thing from a
hash (code ref) and it's w/o threads. I'm trying to reproduce it and see if
it's related.

Regards,
Adi

[p5pRT]

From stas@stason.org

Adrian Enache via RT wrote​:

On Fri, Dec 19, 2003 a.d., Stas Bekman wrote​:

That means that you were able to reproduce it, right?

Unfortunately, yes :-)

For now I'll check if it's something with CLONE .

It looks worse​:

$ perl -mthreads -e 'threads->new(sub{my%h=(1,2);delete $h{1}})->join for 1,2'
Attempt to free unreferenced scalar​: SV 0x4017fe84 at -e line 1.

Fun!

And I have one more below. I'm not 100% if it's the same as yours.

I'm seeing a similar error when deleting a totally different thing from a
hash (code ref) and it's w/o threads. I'm trying to reproduce it and see if
it's related.

I've got it reproduced. I thought it had something to do with XSUB, but a
simple number triggers it​:

#/tmp/delete
package Foo;
use Cwd;
my %objects = (foo => 1);
END { delete $objects{foo} }

package main;
use threads;
threads->new(sub { print "thread started\n";})->detach for (1..1);
sleep 1;

% perl-blead-ithread /tmp/delete
thread started
Attempt to free unreferenced scalar​: SV 0x40179e5c at /tmp/delete line 9.

So it all has to do with cloned package hashes and calling 'delete' on its
members.

__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http​://stason.org/ mod_perl Guide ---> http​://perl.apache.org
mailto​:stas@​stason.org http​://use.perl.org http​://apacheweek.com
http​://modperlbook.org http​://apache.org http​://ticketmaster.com

[p5pRT]

From stas@stason.org

And here is yet another example, this time with XSUB, which I think is a bit
different. I used the Cwd​::fastcwd XSUB just as an example. Any XSUB will do.
What the example does is storing the original CODEREF in a package hash, and
later on restoring it (e.g. if you wanted to redefine Cwd​::fastcwd and then
restore it)​:

#--------------------
package Foo;
use Cwd;
my %objects;
my $s = 'Cwd​::fastcwd';
$objects{$s} = *Cwd​::fastcwd{CODE};

END {
  my $orig_sub = delete $objects{$s};
  #my $orig_sub = $objects{$s};
  #delete $objects{$s};
  no warnings 'redefine';
  *$s = $orig_sub;
}

package main;
use threads;
threads->new(sub { print "thread started\n";})->detach for (1..1);
sleep 1;
#--------------------

If I do​:

  my $orig_sub = delete $objects{$s};

I get​:

Attempt to free unreferenced scalar​:

If I do the same in two steps​:

  my $orig_sub = $objects{$s};
  delete $objects{$s};

everything is fine.

__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http​://stason.org/ mod_perl Guide ---> http​://perl.apache.org
mailto​:stas@​stason.org http​://use.perl.org http​://apacheweek.com
http​://modperlbook.org http​://apache.org http​://ticketmaster.com

[p5pRT]

From stas@stason.org

Had anybody have a chance to look at it? It causes very annoying error reports
with mod_perl 2. It'll be very appreciated if it gets fixed. Thanks!

I've first reported this problem about 5 months ago.

Stas Bekman (via RT) wrote​:

# New Ticket Created by Stas Bekman
# Please include the string​: [perl #24660]
# in the subject line of all future correspondence about this issue.
# <URL​: http​://rt.perl.org​:80/rt3/Ticket/Display.html?id=24660 >

Scalar​::Util​::weaken() proposed to use in the conjunction with the CLONE
function to solve the problem of cloning non-perl objects suffers itself from
cloning.

Here is an example​:

#/tmp/weaken2
package main;
use threads;
use Scalar​::Util;
my $data = "a";
my $obj = \$data;
my $copy = $obj;
Scalar​::Util​::weaken($copy);
threads->new(sub { print "thread started\n";})->detach for (1..1);
sleep 1;

% perl-5.8.1-ithread /tmp/weaken2
thread started
Attempt to free unreferenced scalar SV 0x8154f74.

Oops, a weakened $copy was cloned too, but its guts are now messed
up. What scalar do you ask is unreferenced? It's the magic that was
added by weaken() and cloned during perl_clone, that perl complains about.
Here is a more complex program that proves that with Devel​::Peek​:

#/tmp/weaken
package Foo;

use strict;
use warnings;

use Scalar​::Util;
use Devel​::Peek;

my %objects;
sub new {
my $data = "a";
my $self = bless \$data, __PACKAGE__;
my $copy = $self;
Scalar​::Util​::weaken($copy);

 $objects\{"$self"\} = $copy;
 return $self;

}

sub DESTROY {
warn "DESTROY\n";
}

sub CLONE {
warn "CLONE is called";
for my $key ( keys %objects) {
my $self = delete $objects{$key};
Dump $self;
}
}

package main;
use threads;

my $obj = Foo->new;

threads->new(sub { print "thread started\n";})->detach for (1..1);
sleep 1;

% perl-5.8.1-ithread /tmp/weaken
CLONE is called at /tmp/weaken line 23.
SV = RV(0x8110b2c) at 0x815de1c
REFCNT = 1
FLAGS = (PADBUSY,PADMY,ROK)
RV = 0x8156e1c
SV = PVMG(0x8115e38) at 0x8156e1c
REFCNT = 2
FLAGS = (PADBUSY,PADMY,OBJECT,RMG,POK,pPOK)
IV = 0
NV = 0
PV = 0x8157860 "a"\0
CUR = 1
LEN = 2
MAGIC = 0x8157828
MG_VIRTUAL = &PL_vtbl_backref
MG_TYPE = PERL_MAGIC_backref(<)
MG_FLAGS = 0x02
REFCOUNTED
MG_OBJ = 0x8156e28
SV = PVAV(0x8153494) at 0x8156e28
REFCNT = 1
FLAGS = ()
IV = 0
NV = 0
ARRAY = 0x8157848
FILL = 0
MAX = 3
ARYLEN = 0x0
FLAGS = (REAL)
Elt No. 0
STASH = 0x8156d2c "Foo"
thread started
DESTROY
Attempt to free unreferenced scalar SV 0x8156e28 during global destruction.
DESTROY

As you can see SV 0x8156e28 is​:

   SV = PVAV\(0x8153494\) at 0x8156e28
     REFCNT = 1
     FLAGS = \(\)
     IV = 0
     NV = 0
     ARRAY = 0x8157848
     FILL = 0
     MAX = 3
     ARYLEN = 0x0
     FLAGS = \(REAL\)
     Elt No\. 0

Any chance we can change perl to call Devel​::Peek​::Dump when producing​:

Attempt to free unreferenced scalar SV 0x8154f74\.

would save us a lot of time, trying to figure out what SV went lost.

As Liz has pointed out​:

The problem with the weakened scalar does not exist in 5.8.0, but does exist
in 5.8.1 and 5.8.2. So it must have happened somewhere between 5.8.0 and 5.8.1.

perl-5.8.1-ithread -V
Summary of my perl5 (revision 5.0 version 8 subversion 1) configuration​:
Platform​:
osname=linux, osvers=2.4.21-0.18mdkcustom, archname=i686-linux-thread-multi
uname='linux rabbit.stason.org 2.4.21-0.18mdkcustom #6 mon jun 16
16​:26​:34 est 2003 i686 unknown unknown gnulinux '
config_args='-des -Dprefix=/home/stas/perl/5.8.1-ithread -Dusethreads
-Doptimize=-g -Duseshrplib -Dusedevel'
hint=recommended, useposix=true, d_sigaction=define
usethreads=define use5005threads=undef useithreads=define
usemultiplicity=define
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler​:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS
-DDEBUGGING -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
optimize='-g',
cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING
-fno-strict-aliasing -I/usr/local/include -I/usr/include/gdbm'
ccversion='', gccversion='3.3.1 (Mandrake Linux 9.2 3.3.1-2mdk)',
gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries​:
ld='cc', ldflags =' -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib
libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
libc=/lib/libc-2.3.2.so, so=so, useshrplib=true, libperl=libperl.so
gnulibc_version='2.3.2'
Dynamic Linking​:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic
-Wl,-rpath,/home/stas/perl/5.8.1-ithread/lib/5.8.1/i686-linux-thread-multi/CORE'
cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'

Characteristics of this binary (from libperl)​:
Compile-time options​: DEBUGGING MULTIPLICITY USE_ITHREADS USE_LARGE_FILES
PERL_IMPLICIT_CONTEXT
Built under linux
Compiled at Oct 9 2003 11​:57​:40
%ENV​:
PERLDOC_PAGER="less -R"
@​INC​:
/home/stas/perl/5.8.1-ithread/lib/5.8.1/i686-linux-thread-multi
/home/stas/perl/5.8.1-ithread/lib/5.8.1
/home/stas/perl/5.8.1-ithread/lib/site_perl/5.8.1/i686-linux-thread-multi
/home/stas/perl/5.8.1-ithread/lib/site_perl/5.8.1
/home/stas/perl/5.8.1-ithread/lib/site_perl
.

--
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http​://stason.org/ mod_perl Guide ---> http​://perl.apache.org
mailto​:stas@​stason.org http​://use.perl.org http​://apacheweek.com
http​://modperlbook.org http​://apache.org http​://ticketmaster.com

[p5pRT]

From @smpeters

[stas - Thu Apr 22 12​:43​:19 2004]​:

Had anybody have a chance to look at it? It causes very annoying error
reports
with mod_perl 2. It'll be very appreciated if it gets fixed. Thanks!

I've first reported this problem about 5 months ago.

Stas Bekman (via RT) wrote​:

# New Ticket Created by Stas Bekman
# Please include the string​: [perl #24660]
# in the subject line of all future correspondence about this issue.
# <URL​: http​://rt.perl.org​:80/rt3/Ticket/Display.html?id=24660 >

Scalar​::Util​::weaken() proposed to use in the conjunction with the
CLONE
function to solve the problem of cloning non-perl objects suffers
itself from
cloning.

Here is an example​:

#/tmp/weaken2
package main;
use threads;
use Scalar​::Util;
my $data = "a";
my $obj = \$data;
my $copy = $obj;
Scalar​::Util​::weaken($copy);
threads->new(sub { print "thread started\n";})->detach for (1..1);
sleep 1;

% perl-5.8.1-ithread /tmp/weaken2
thread started
Attempt to free unreferenced scalar SV 0x8154f74.

Oops, a weakened $copy was cloned too, but its guts are now messed
up. What scalar do you ask is unreferenced? It's the magic that was
added by weaken() and cloned during perl_clone, that perl complains
about.
Here is a more complex program that proves that with Devel​::Peek​:

#/tmp/weaken
package Foo;

use strict;
use warnings;

use Scalar​::Util;
use Devel​::Peek;

my %objects;
sub new {
my $data = "a";
my $self = bless \$data, __PACKAGE__;
my $copy = $self;
Scalar​::Util​::weaken($copy);

 $objects\{"$self"\} = $copy;
 return $self;

}

sub DESTROY {
warn "DESTROY\n";
}

sub CLONE {
warn "CLONE is called";
for my $key ( keys %objects) {
my $self = delete $objects{$key};
Dump $self;
}
}

package main;
use threads;

my $obj = Foo->new;

threads->new(sub { print "thread started\n";})->detach for (1..1);
sleep 1;

% perl-5.8.1-ithread /tmp/weaken
CLONE is called at /tmp/weaken line 23.
SV = RV(0x8110b2c) at 0x815de1c
REFCNT = 1
FLAGS = (PADBUSY,PADMY,ROK)
RV = 0x8156e1c
SV = PVMG(0x8115e38) at 0x8156e1c
REFCNT = 2
FLAGS = (PADBUSY,PADMY,OBJECT,RMG,POK,pPOK)
IV = 0
NV = 0
PV = 0x8157860 "a"\0
CUR = 1
LEN = 2
MAGIC = 0x8157828
MG_VIRTUAL = &PL_vtbl_backref
MG_TYPE = PERL_MAGIC_backref(<)
MG_FLAGS = 0x02
REFCOUNTED
MG_OBJ = 0x8156e28
SV = PVAV(0x8153494) at 0x8156e28
REFCNT = 1
FLAGS = ()
IV = 0
NV = 0
ARRAY = 0x8157848
FILL = 0
MAX = 3
ARYLEN = 0x0
FLAGS = (REAL)
Elt No. 0
STASH = 0x8156d2c "Foo"
thread started
DESTROY
Attempt to free unreferenced scalar SV 0x8156e28 during global
destruction.
DESTROY

As you can see SV 0x8156e28 is​:

   SV = PVAV\(0x8153494\) at 0x8156e28
     REFCNT = 1
     FLAGS = \(\)
     IV = 0
     NV = 0
     ARRAY = 0x8157848
     FILL = 0
     MAX = 3
     ARYLEN = 0x0
     FLAGS = \(REAL\)
     Elt No\. 0

Any chance we can change perl to call Devel​::Peek​::Dump when
producing​:

Attempt to free unreferenced scalar SV 0x8154f74\.

would save us a lot of time, trying to figure out what SV went lost.

As Liz has pointed out​:

The problem with the weakened scalar does not exist in 5.8.0, but
does exist
in 5.8.1 and 5.8.2. So it must have happened somewhere between
5.8.0 and 5.8.1.

I'm no longer seeing this issue on perl-5.8.7. Is anyone able to
recreate with a recent Perl on any platform?

[p5pRT]

From @iabyn

On Sun, Oct 16, 2005 at 09​:18​:53AM -0700, Steve Peters via RT wrote​:

I'm no longer seeing this issue on perl-5.8.7. Is anyone able to
recreate with a recent Perl on any platform?

Appears to have been fixed somewhere between 21874 and 21978

--
I've often wanted to drown my troubles, but I can't get my wife to go
swimming.