Data::Dumper + threads + recursion = segfault #6782

p5pRT · 2003-09-22T17:01:18Z

Migrated from rt.perl.org#23965 (status was 'resolved')

Searchable as RT23965$

p5pRT · 2003-09-22T17:01:18Z

From DavidBuckley@bigfoot.com

Created by DavidBuckley@bigfoot.com

The following code illustrates the issue nicely:

use threads;
use threads::shared;
use Data::Dumper;
our $a : shared;
$a = \$a;
print Data::Dumper::Dumper( $a );

This produces a short pause, then a segfault. I'm guessing it's due
to issues with the way references to shared objects are implemented
(at least on Linux):

use threads;
use threads::shared;
our $a : shared = &share( [] );
print $a;
print $a;

Gives:

ARRAY(0x813bbc4)ARRAY(0x813babc)

Which, of course, is going to confuse anything that checks
cross-references using the reference itself.

There appears to be no obvious way of checking equivalence of shared
references; presumably they're implemented in such a way that they're
not really references to a specific object, but floaty references to
the shared version. I'm currently avoiding the segfault by specifying
a maximum recursive depth.

Perl Info


Flags:
    category=core
    severity=high

This perlbug was built using Perl v5.8.0 - Sun Jun 16 02:13:30 BST 2002
It is being executed now by  Perl v5.8.0 - Thu Sep 11 20:57:21 EST 2003.

Site configuration information for perl v5.8.0:

Configured by Debian Project at Thu Sep 11 20:57:21 EST 2003.

Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration:
  Platform:
    osname=linux, osvers=2.4.21-xfs+ti1211, archname=i386-linux-thread-multi
    uname='linux kosh 2.4.21-xfs+ti1211 #1 sat jul 12 10:35:04 est 2003 i686 gnulinux '
    config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=i386-linux -Dprefix=/usr -Dprivlib=/usr/share/perl/5.8.0 -Darchlib=/usr/lib/perl/5.8.0 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.8.0 -Dsitearch=/usr/local/lib/perl/5.8.0 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Uusesfio -Uusenm -Duseshrplib -Dlibperl=libperl.so.5.8.0 -Dd_dosuid -des'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O3',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing'
    ccversion='', gccversion='3.3.2 20030908 (Debian prerelease)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lgdbm -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=/lib/libc-2.3.2.so, so=so, useshrplib=true, libperl=libperl.so.5.8.0
    gnulibc_version='2.3.2'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'

Locally applied patches:
    


@INC for perl v5.8.0:
    /etc/perl
    /usr/local/lib/perl/5.8.0
    /usr/local/share/perl/5.8.0
    /usr/lib/perl5
    /usr/share/perl5
    /usr/lib/perl/5.8.0
    /usr/share/perl/5.8.0
    /usr/local/lib/site_perl
    .


Environment for perl v5.8.0:
    HOME=/home/bucko
    LANG=C
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/bucko/bin:/usr/local/jdk1.2.2/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games
    PERL_BADLANG (unset)
    SHELL=/bin/bash

p5pRT · 2003-09-23T04:27:25Z

From perl_dummy@bloodgate.com

-----BEGIN PGP SIGNED MESSAGE-----

Moin,

# New Ticket Created by David Buckley
# Please include the string: [perl #23965]
# in the subject line of all future correspondence about this issue.
# <URL: http://rt.perl.org/rt2/Ticket/Display.html?id=23965 >

use threads;
use threads::shared;
our $a : shared = &share( [] );
print $a;
print $a;

Gives:

ARRAY(0x813bbc4)ARRAY(0x813babc)

Interesting, does it give a different value each time you print it, or does it
alternate it between these two?

Best wishes,

Tels

- --
Signed on Tue Sep 23 06:27:05 2003 with key 0x93B84C15.
Visit my photo gallery at http://bloodgate.com/photos/
PGP key on http://bloodgate.com/tels.asc or per email.

"Where shall I put you? Under H, like Hot, Sexy Mama?"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2-rc1-SuSE (GNU/Linux)
Comment: When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl.

iQEVAwUBP2/L3ncLPEOTuEwVAQECkQf+Jsi/3oiKKK4uox1Ckm/PUypxW+UuU6cI
iymStV9uEQcBTZ6JtYJD+ueEKhzCTNyj0emdpGpr2RT1zTmyg9Vy/V9Lmsm9AUIH
raKL9sr/3WYfralXV6e9omIoOohdumGHa1yetJXGy+ZF6hLiGiDIBiiuIklkAHzW
bL25FYUjZtAi4MYCKqgvi6+s6zw2G8uP3PEYYQNp0J/9X6TAoc2SA0cE5P74kHgu
lNBlt9i5R2/nQ8pzI1EW41wegK0OSLpDvnbP9wh+21tnD+YFLxbjO8MWIxJW1AgD
XivI8gWXxOoMTWDUbuxCYJZqgTQwxmMmJIitGeVPRWCV4o9Bna6x8A==
=t5u7
-----END PGP SIGNATURE-----

p5pRT · 2003-09-23T12:48:22Z

From DavidBuckley@bigfoot.com

On 23 Sep 2003, Tels wrote:

-----BEGIN PGP SIGNED MESSAGE-----

Moin,

# New Ticket Created by David Buckley
# Please include the string: [perl #23965]
# in the subject line of all future correspondence about this issue.
# <URL: http://rt.perl.org/rt2/Ticket/Display.html?id=23965 >

use threads;
use threads::shared;
our $a : shared = &share( [] );
print $a;
print $a;

Gives:

ARRAY(0x813bbc4)ARRAY(0x813babc)

Interesting, does it give a different value each time you print it, or does it
alternate it between these two?

<snip>

It seems to just alternate. Different threads can, however, appear to
read different vales, as shown here:

[11:34:29] bucko(tank) ~$ perl -Mthreads -Mthreads::shared -e 'my $a : shared = &share({}); for(1..2){async{for(1..5) {print threads->tid()."$a\n"}}}sleep 2'
1HASH(0x8272960)
1HASH(0x8203778)
1HASH(0x8272960)
1HASH(0x8203778)
1HASH(0x8272960)
2HASH(0x82dabd8)
2HASH(0x827b3c0)
2HASH(0x82dabd8)
2HASH(0x827b3c0)
2HASH(0x82dabd8)

(Apologies for messy code)

My original code was using 5 processing threads, hence giving a
selection of 10 values (presumably), which is what gave me the
impression it was just allocating a new one each time.

Oddly enough, this kinda crushes my theory on Data::Dumper hitting an
infinate loop.

Another interesting case:

[11:40:10] bucko(tank) ~$ perl -Mthreads -Mthreads::shared -e 'my $a :
shared; $a = \$a; for(1..5){print "$a\n" for (1..2); $a = $$a}'
SCALAR(0x81f2db8)
SCALAR(0x815054c)
SCALAR(0x813bc0c)
SCALAR(0x81f2db8)
SCALAR(0x815054c)
SCALAR(0x813bc0c)
SCALAR(0x81f2db8)
SCALAR(0x815054c)
SCALAR(0x813bc0c)
SCALAR(0x81f2db8)

Creating a third value, and note how they run cyclically still. This
could imply it's to do with whenever the variable gets dereferenced.

Finally, this one appears to have some /immense/ number of possible
values:

[11:43:56] bucko(tank) ~$ perl -Mthreads -Mthreads::shared -e 'my $a : shared = &share({}); $a->{a} = $a; $a->{b} = "f"; for(1..5){print "$_ $a $a->{a} $a->{b}\n" for (1..3); $a = $a->{a}}'
1 HASH(0x81f2fa0) HASH(0x81f306c) f
2 HASH(0x813babc) HASH(0x81f3084) f
3 HASH(0x81f309c) HASH(0x81f303c) f
1 HASH(0x81f2fa0) HASH(0x81f2fb8) f
2 HASH(0x81f309c) HASH(0x814f2c4) f
3 HASH(0x81f3030) HASH(0x81f3060) f
1 HASH(0x81f2fa0) HASH(0x81f2ff4) f
2 HASH(0x81f3030) HASH(0x81f30b4) f
3 HASH(0x813bc0c) HASH(0x81f2fac) f
1 HASH(0x81f2fa0) HASH(0x81f3054) f
2 HASH(0x813bc0c) HASH(0x813babc) f
3 HASH(0x81f3018) HASH(0x81f3090) f
1 HASH(0x81f2fa0) HASH(0x81f303c) f
2 HASH(0x81f3018) HASH(0x81f309c) f
3 HASH(0x81f3078) HASH(0x81f3084) f

Note how there however /are/ patterns in the apparent mess.

A simple hash based unique value finder tells me there's 12 values here,
if you don't keep dereferencing $a into itself. Dereferencing $a gives
"number of dereferences plus 2" ie 10002 possible values, in the
following code:

my %count;
my $a : shared = &share({});
$a->{a} = $a;
for(1..10000) {
$count{"$a"}++;
$count{"$a->{a}"}++;
$a = $a->{a};
}
print scalar keys %count;
print "\n";

(Run through perl -Mthreads -Mthreads::shared)

Bizarrely, when I reformatted the code like this:

my %count;
my $a : shared = &share({});
$a->{a} = $a;
for(1..10000) {
my $temp = "$a";
print "$_ $temp ";
$count{ $temp }++;
$temp = "$a->{a}";
print "$temp\n";
$count{ $temp }++;
$a = $a->{a};
}
print scalar keys %count;
print "\n";

I got 20000 values. Removing the my qualifier on $temp does nothing
here, so it's presumably not to do with variable allocation. Removing
the print statements also does nothing. Reformatting the body of the
loop like this:

$count{ $temp = "$a" }++;
$count{ $temp = "$a->{a}" }++;
$a = $a->{a};

Brings me back to 10002, however.

Perhaps scarily, the following body gives 20 unique values for the same
10000 loops:

$count{ $temp = "$a" }++;
$count{ $temp = "$a->{a}" }++;
$count{ $temp = "$a->{a}->{a}" }++;
$a = $a->{a};

Even more worrying, the following body gives /11/ unique values:

$count{ $temp = "$a" }++;
$count{ $temp = "$a->{a}" }++;
$count{ $temp = "$a->{a}->{a}" }++;
$a = $a->{a};
$a = $a->{a};

Adding a further deference brings it up to 14. Combining the two lines
into one gives 1679 values. Consistently.

So, yes, I think I'll leave this up to someone who actually knows what
the hell is going on here. :)

Apologies for the somewhat inconsistent code etc. I was basically just
cut-n-paste playing around with the script, so some bits from elsewhere
got left in bits they shouldn't have been. I'd test on other systems,
but I only have Linux to play with)

HTH

--
bucko

p5pRT · 2003-09-23T16:09:39Z

From perl_dummy@bloodgate.com

-----BEGIN PGP SIGNED MESSAGE-----

Moin David,

It seems to just alternate. Different threads can, however, appear to
read different vales, as shown here:

Strange. But an idea particle suddenly hit me while reading this. There _was_
a memory leak with shared arrays in threads (or something along these lines,
threads just scare me) and that should be fixed in 5.8.1-to-be. You could try
RC5 from

http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2003-09/msg01289.html

there or the latest bleadperl. If you could give these a spin and see if the
issue persist, than that would be easier than to trying tracking down the
mess.

Best wishes,

Te"Armchair Debugger Extraordinaire"ls

- --
Signed on Tue Sep 23 18:07:21 2003 with key 0x93B84C15.
Visit my photo gallery at http://bloodgate.com/photos/
PGP key on http://bloodgate.com/tels.asc or per email.

"Naturally the parameter and boundary of their respective position and
magnitude are naturally determinable up to the limits of possible
measurement as stated by the general quantum hypothesis and Heisenberg's
uncertainty principle, but this indeterminacy in precise value is not a
consequence of quantum uncertainty. What this illustrates is that in
relation to indeterminacy in precise physical magnitude, the micro and
macroscopic are inextricably linked, both being a part of the same parcel,
rather than just a case of the former underlying and contributing to the
latter." -- Peter Lynd

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2-rc1-SuSE (GNU/Linux)
Comment: When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl.

iQEVAwUBP3BwgHcLPEOTuEwVAQHD1Af+NoJPeXhQoKXHAdurrCgC2VVocXfnChnF
z7EDGsTBDmfuD71QNWpbFwoElVPJrluqucnvsKoVKBsBvhV4aVvaQed4wgzt2aAa
IGZddUC7jCns5HfhgeR/DC5kuGy/yJfJ20zQ/PkuvMbb29NpYvXKxHHK6CKvVQtB
aGM0RqnuKXxlcPeo93C1Y2zgktxU+LI9esz1mIUbzbizX6AWtaZap9cjUGLqHisu
I5YZXo1/+w5DrRZVFk6I0Yy4yzOzhVmMlV+mE7R+aHsYI41CA0i0KSkIqRkwFpDN
aT/phwrfnCbmVawGBhSELvelETqqsjhsHYQ0VKhgWoQMTwaMvGGwPg==
=c1jD
-----END PGP SIGNATURE-----

p5pRT · 2003-09-23T19:38:52Z

From DavidBuckley@bigfoot.com

On 23 Sep 2003, Tels wrote:

-----BEGIN PGP SIGNED MESSAGE-----

Moin David,

It seems to just alternate. Different threads can, however, appear to
read different vales, as shown here:

Strange. But an idea particle suddenly hit me while reading this. There _was_
a memory leak with shared arrays in threads (or something along these lines,
threads just scare me) and that should be fixed in 5.8.1-to-be. You could try
RC5 from
http&#8203;://www\.xray\.mpe\.mpg\.de/mailing\-lists/perl5\-porters/2003\-09/msg01289\.html
there or the latest bleadperl. If you could give these a spin and see if the
issue persist, than that would be easier than to trying tracking down the
mess.

Just did so, but the bug remains. I'm attaching configuration info
below. (I used the bz2 file at the top of the message)

Inspired by your comments on the memory leak, I did some stats. Using
the "hash counter" thing I gave at the end of the last message, I've
brought Perl up to lots of memoy usage by recursively dereferencing a
shared hash, so it probably /is/ leaking memory.

Sample low-iteration run:

[18:49:29] bucko(tank) ~/perl_5.8.1-RC5/bin$ ./perl -Mthreads -Mthreads::shared
my %count;
my $a : shared = &share({});
$a->{a} = $a;
for(1..10000) {
$count{"$a"}++;
$count{"$a->{a}"}++;
$a = $a->{a};
}
print scalar keys %count;
print "\n";
10002

Summary of my perl5 (revision 5.0 version 8 subversion 1) configuration:
Platform:
osname=linux, osvers=2.4.21, archname=i686-linux-thread-multi-ld
uname='linux tank 2.4.21 #1 fri jul 25 00:22:37 bst 2003 i686 gnulinux '
config_args=''
hint=recommended, useposix=true, d_sigaction=define
usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=define
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O3',
cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -fno-strict-aliasing -I/usr/local/include'
ccversion='', gccversion='3.3.2 20030812 (Debian prerelease)', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long', ivsize=4, nvtype='long double', nvsize=12, Off_t='off_t', lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib
libs=-lnsl -ldb -ldl -lm -lcrypt -lutil -lpthread -lc
perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
libc=/lib/libc-2.3.2.so, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.3.2'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic'
cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'

Characteristics of this binary (from libperl):
Compile-time options: MULTIPLICITY USE_ITHREADS USE_LONG_DOUBLE USE_LARGE_FILES PERL_IMPLICIT_CONTEXT
Locally applied patches:
RC5
Built under linux
Compiled at Sep 23 2003 18:39:06
@INC:
/home/bucko/perl_5.8.1-RC5/lib/5.8.1/i686-linux-thread-multi-ld
/home/bucko/perl_5.8.1-RC5/lib/5.8.1
/home/bucko/perl_5.8.1-RC5/lib/site_perl/5.8.1/i686-linux-thread-multi-ld
/home/bucko/perl_5.8.1-RC5/lib/site_perl/5.8.1
/home/bucko/perl_5.8.1-RC5/lib/site_perl
.

--
bucko

p5pRT · 2003-09-24T00:09:43Z

From @iabyn

On Mon, Sep 22, 2003 at 05:01:19PM -0000, David Buckley wrote:

use threads;
use threads::shared;
use Data::Dumper;
our $a : shared;
$a = \$a;
print Data::Dumper::Dumper( $a );

This produces a short pause, then a segfault.

The segfault is due to stack or memory exhaustion caused by infinite
recursion. The problem is due to the 'proxy' per-thread variables
failing to detect a loop caused by mg_get(). To explain further:
each shared SV lives in a special shared area; each thread
has a stub 'proxy' SV with magic. When the the proxy SV is accessed, its
magic is called, which locks the real SV, gets/sets its real value, and
updates the proxy SV to reflect that value.

In the case of $a = \$a, the real SV is 'correct': it's an RV which points
to itself. When the proxy (ie the $a in the main thread) is accessed,
mg_get is called; it realises that the real $a is an RV, so it creates
a tmp RV with shared magic and makes the proxy point to this.
After the mg_get, you follow through to the thing the proxy $a is pointing
to. This is the tmp SV with the shared magic. So you call mg_get on it,
which repreats the process ad nauseum.

Note that this is a bug in the shared var implementation, not in
Data::Dumper. The following code demonstrates:

use threads;
use threads::shared;

my $a : shared;

$a = \$a;

my $x = $a;
for (1..10) {
print "$x\n";
$x = $$x;
}

outputs:

scalar(0x81a3b08)
scalar(0x81bbf00)
scalar(0x8213930)
scalar(0x81c41b8)
scalar(0x81bbeb8)
scalar(0x8207fd4)
scalar(0x8211f14)
scalar(0x8220c3c)
scalar(0x81bacdc)
scalar(0x8258a10)

I don't know whether the other parts of the bug report are related.

And no, I don't know how to fix it.
(Well, I could see maybe how to fix it for the trivial $a=\$a case,
but not for more general self-referential data structures.)

Dave.

--
"There's something wrong with our bloody ships today, Chatfield."
Admiral Beatty at the Battle of Jutland, 31st May 1916.

p5pRT · 2005-07-14T08:58:07Z

From @schwern

[davem - Tue Sep 23 17:09:43 2003]:

On Mon, Sep 22, 2003 at 05:01:19PM -0000, David Buckley wrote:

use threads;
use threads::shared;
use Data::Dumper;
our $a : shared;
$a = \$a;
print Data::Dumper::Dumper( $a );

This produces a short pause, then a segfault.

The segfault is due to stack or memory exhaustion caused by infinite
recursion. The problem is due to the 'proxy' per-thread variables
failing to detect a loop caused by mg_get().

This is still an issue in bleadperl@25129.

p5pRT · 2006-01-08T19:40:56Z

From @iabyn

On Thu, Jul 14, 2005 at 01:58:08AM -0700, Michael G Schwern via RT wrote:

[davem - Tue Sep 23 17:09:43 2003]:

On Mon, Sep 22, 2003 at 05:01:19PM -0000, David Buckley wrote:

use threads;
use threads::shared;
use Data::Dumper;
our $a : shared;
$a = \$a;
print Data::Dumper::Dumper( $a );

This produces a short pause, then a segfault.

The segfault is due to stack or memory exhaustion caused by infinite
recursion. The problem is due to the 'proxy' per-thread variables
failing to detect a loop caused by mg_get().

This is still an issue in bleadperl@25129.

Fixed in bleed by change #26695

--
A walk of a thousand miles begins with a single step...
then continues for another 1,999,999 or so.

p5pRT · 2006-01-08T19:41:31Z

@iabyn - Status changed from 'open' to 'resolved'

p5pRT closed this as completed Jan 8, 2006

p5pRT added Severity High distro-Linux type-core labels Oct 18, 2019

p5pRT mentioned this issue Oct 18, 2019

threads sort crashes with sort subroutine (but not with sort block) #7372

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Data::Dumper + threads + recursion = segfault #6782

Data::Dumper + threads + recursion = segfault #6782

p5pRT commented Sep 22, 2003

p5pRT commented Sep 22, 2003

p5pRT commented Sep 23, 2003

p5pRT commented Sep 23, 2003

p5pRT commented Sep 23, 2003

p5pRT commented Sep 23, 2003

p5pRT commented Sep 24, 2003

p5pRT commented Jul 14, 2005

p5pRT commented Jan 8, 2006

p5pRT commented Jan 8, 2006

Data::Dumper + threads + recursion = segfault #6782

Data::Dumper + threads + recursion = segfault #6782

Comments

p5pRT commented Sep 22, 2003

p5pRT commented Sep 22, 2003

From DavidBuckley@bigfoot.com

Created by DavidBuckley@bigfoot.com

p5pRT commented Sep 23, 2003

From perl_dummy@bloodgate.com

p5pRT commented Sep 23, 2003

From DavidBuckley@bigfoot.com

p5pRT commented Sep 23, 2003

From perl_dummy@bloodgate.com

p5pRT commented Sep 23, 2003

From DavidBuckley@bigfoot.com

p5pRT commented Sep 24, 2003

From @iabyn

p5pRT commented Jul 14, 2005

From @schwern

p5pRT commented Jan 8, 2006

From @iabyn

p5pRT commented Jan 8, 2006