Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Safe.pm sort {} bug with -Dusethreads #9557

Closed
p5pRT opened this issue Nov 6, 2008 · 48 comments
Closed

Safe.pm sort {} bug with -Dusethreads #9557

p5pRT opened this issue Nov 6, 2008 · 48 comments
Labels

Comments

@p5pRT
Copy link
Collaborator

@p5pRT p5pRT commented Nov 6, 2008

Migrated from rt.perl.org#60374 (status was 'resolved')

Searchable as RT60374$

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 6, 2008

From badalex@gmail.com

When using a threading perl and you reval an anon sub that contains a sort,
a/$b loose their values. Both 5.8.8 and 5.10.0 seem to be affected.

(see attached perl.v for perl -V output)
-Dusethreads
perl tsafe.pl
debug​:
debug​:
debug​:
debug​:
debug​:
debug​:
debug​:
debug​:
debug​: 5 4 3 2 1

without -Dusethreads
perl1 tsafe.pl
debug​: 4
debug​: 2
debug​: 2
debug​: 3
debug​: 1
debug​: 1 2 3 4 5

tsafe.pl

use Safe;

my $safe = Safe->new('PLPerl');
$safe->permit_only('​:default');
$safe->permit(qw(sort));
$safe->share(qw(&debug));

sub debug {
  print "debug​: ". (shift) . "\n";
}

my $func = $safe->reval(<<'z');
sub {
my @​c = sort { debug("$b"); "$a" <=> $b } qw(5 4 3 2 1);
debug(join(" ", @​c));
return;
}
z

#my $f = eval 'package PLPerl; sub { @​_=(); $func->(); }';
$func->();

die $@​ if($@​);

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 6, 2008

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 6, 2008

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 6, 2008

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 6, 2008

From badalex@gmail.com

Andrew Gierth <andrew@​tao11.riddles.org.uk> managed to track it down
his explanation is below​:

If it helps any, I've tracked down in the perl guts exactly why this
happens​:

cop.h​:

struct cop {
  BASEOP
  char * cop_label; /* label for this construct */
#ifdef USE_ITHREADS
  char * cop_stashpv; /* package line was compiled in */
  char * cop_file; /* file name the following line # is from */
#else
  HV * cop_stash; /* package line was compiled in */
  GV * cop_filegv; /* file the following line # is from */
#endif
  U32 cop_seq; /* parse sequence number */
  I32 cop_arybase; /* array base this line was compiled with */
  line_t cop_line; /* line # of this command */
  SV * cop_warnings; /* lexical warnings bitmask */
  SV * cop_io; /* lexical IO defaults */
};

A COP in perl is a control operation, basically a compiled statement,
and the pointer to the current COP is used to determine all the
lexical state, including the current package. pp_sort uses
CopSTASH(PL_curcop) to get the package stash (symbol table) in order
to locate the $a and $b variables in it.

Notice, though, that without ithreads, the COP points directly to the
stash, but with ithreads, it points instead to the _name_ of the stash
(e.g. "main"). The problem arises because with Safe in use, the
package created by Safe to use as a container _thinks_ that its name
is "main" even though it's not, so the COPs compiled inside it point
to the name "main" rather than to the real name of the container.

So with ithreads enabled, pp_sort looks up the package stash by name,
gets the "main" package rather than the safe container, and creates
$main​::a and $main​::b to store the comparison values in. But the
compiled comparison block has its own references to the variables
which refers to the correct stash, so it all goes Horribly Wrong at
that point.

So there are three factors involved​:

1) the change in layout of COP with ithreads enabled
2) the fact that Safe changes the internally-seen name of a package
3) any operation that relies on CopSTASH(PL_curcop) (I can only find a
few​: sort, reset, and bless) will then behave incorrectly

However, I have no idea why Perl has this difference between threaded
and non-threaded code.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 6, 2008

badalex@gmail.com - Status changed from 'new' to 'open'

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 8, 2008

From badalex@gmail.com

Andrew Gierth <andrew@​tao11.riddles.org.uk> managed to track it down
his explanation is below​:

If it helps any, I've tracked down in the perl guts exactly why this
happens​:

cop.h​:

struct cop {
  BASEOP
  char * cop_label; /* label for this construct */
#ifdef USE_ITHREADS
  char * cop_stashpv; /* package line was compiled in */
  char * cop_file; /* file name the following line # is from */
#else
  HV * cop_stash; /* package line was compiled in */
  GV * cop_filegv; /* file the following line # is from */
#endif
  U32 cop_seq; /* parse sequence number */
  I32 cop_arybase; /* array base this line was compiled with */
  line_t cop_line; /* line # of this command */
  SV * cop_warnings; /* lexical warnings bitmask */
  SV * cop_io; /* lexical IO defaults */
};

A COP in perl is a control operation, basically a compiled statement,
and the pointer to the current COP is used to determine all the
lexical state, including the current package. pp_sort uses
CopSTASH(PL_curcop) to get the package stash (symbol table) in order
to locate the $a and $b variables in it.

Notice, though, that without ithreads, the COP points directly to the
stash, but with ithreads, it points instead to the _name_ of the stash
(e.g. "main"). The problem arises because with Safe in use, the
package created by Safe to use as a container _thinks_ that its name
is "main" even though it's not, so the COPs compiled inside it point
to the name "main" rather than to the real name of the container.

So with ithreads enabled, pp_sort looks up the package stash by name,
gets the "main" package rather than the safe container, and creates
$main​::a and $main​::b to store the comparison values in. But the
compiled comparison block has its own references to the variables
which refers to the correct stash, so it all goes Horribly Wrong at
that point.

So there are three factors involved​:

1) the change in layout of COP with ithreads enabled
2) the fact that Safe changes the internally-seen name of a package
3) any operation that relies on CopSTASH(PL_curcop) (I can only find a
  few​: sort, reset, and bless) will then behave incorrectly

However, I have no idea why Perl has this difference between threaded
and non-threaded code.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 6, 2009

From @timbunce

Any news of progress with this bug?

Meanwhile, I have a workaround​: explicitly share the globs for a and b. In other words, add *a
and * b to the arguments of the share() method​:

$safe->share(qw(&debug *a *b));

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 7, 2009

From badalex@gmail.com

On Tue, Oct 6, 2009 at 13​:55, Tim Bunce via RT
<perlbug-followup@​perl.org> wrote​:

Any news of progress with this bug?

Nope still broke for me on 5.10.1

Here are the workaround I know about​:
1) dont use $a and $b in sort, use @​_
2) your way of adding *a and *b, but that only works as long as the
package stays the same...
3) declare the package at the top of reval, i.e.
  my $safe = Safe->new('PLPerl');
  $safe->permit(qw(sort));
  my $func = $safe->reval(<<'eval');
  package PLPerl;
  sub {
  sort { "$a" cmp "$b" } qw(works now);
  }
  eval

  $func->();

4) fix perl :) as mentioned in the previous message

Meanwhile, I have a workaround​: explicitly share the globs for a and b. In other words, add *a
and * b to the arguments of the share() method​:

$safe->share(qw(&debug *a *b));

Yeah as noted above that should work. Unless you ever sort inside a
different package than you gave to Safe->new() (see the below test
case)

use Safe();

my $safe = Safe->new('PLPerl');
$safe->permit_only('​:default');
$safe->permit(qw(sort));
$safe->share(qw(&PLPerl​::debug *a *b));

my $func = $safe->reval(<<'z');
sub {
package PLPrl;
my @​c = sort { PLPerl​::debug("$a ". $b); "$a" <=> $b } qw(5 4 3 2 1);
PLPerl​::debug(join(" ", @​c));
return;
}
z

$func->();

package PLPerl;
sub debug {
  print "debug​: ". (shift) . "\n";
}

$ perl tsafe.pl
debug​:
debug​:
debug​:
debug​:
debug​:
debug​:
debug​:
debug​:
debug​: 5 4 3 2 1

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 7, 2009

From @rgarcia

2009/10/6 Tim Bunce via RT <perlbug-followup@​perl.org>​:

Any news of progress with this bug?

Meanwhile, I have a workaround​: explicitly share the globs for a and b. In other words, add *a
and * b to the arguments of the share() method​:

$safe->share(qw(&debug *a *b));

I don't think that *a and *b should be added to the default sharelist
of Safe. Actually the only glob there is *_, and I think it should
remain as is.

The bug is most likely in the sort implementation, which is
notoriously thread-unsafe when it invokes sort subs.

In short, my advice would be : don't use sort() at all with a threaded perl.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 7, 2009

From david@kineticode.com

On Oct 7, 2009, at 8​:22 AM, Rafael Garcia-Suarez wrote​:

The bug is most likely in the sort implementation, which is
notoriously thread-unsafe when it invokes sort subs.

In short, my advice would be : don't use sort() at all with a
threaded perl.

Say what? Sure, on current releases, but surely that should be fixed.
There are more and more threaded Perls every day…

David

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 7, 2009

From badalex@gmail.com

On Wed Oct 07 08​:22​:57 2009, rgs@​consttype.org wrote​:

The bug is most likely in the sort implementation, which is
notoriously thread-unsafe when it invokes sort subs.

In short, my advice would be : don't use sort() at all with a threaded
perl.

Um... you do realize its just a threaded *build* of perl. Nothing here
is actually threaded or even using threads... Right?

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 7, 2009

From @rgarcia

2009/10/7 David E. Wheeler <david@​kineticode.com>​:

On Oct 7, 2009, at 8​:22 AM, Rafael Garcia-Suarez wrote​:

The bug is most likely in the sort implementation, which is
notoriously thread-unsafe when it invokes sort subs.

In short, my advice would be : don't use sort() at all with a threaded
perl.

Say what? Sure, on current releases, but surely that should be fixed. There
are more and more threaded Perls every day…

Even if we manage someone to fix threading, this will slow down
sort(). Threading also has a non-negligeable performance impact on
memory and speed of every perl program.

I think we should advice people in INSTALL to build with threading
only if they really need it, and tell OS vendors to disable threading
on their system perls.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 7, 2009

From david@kineticode.com

On Oct 7, 2009, at 10​:24 AM, Rafael Garcia-Suarez wrote​:

Even if we manage someone to fix threading, this will slow down
sort(). Threading also has a non-negligeable performance impact on
memory and speed of every perl program.

I think we should advice people in INSTALL to build with threading
only if they really need it, and tell OS vendors to disable threading
on their system perls.

Snow Leopard ships with a threaded Perl. Hell, I think Leopard did.
That cat might have left the bag.

Best,

David

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 8, 2009

From @jandubois

On Wed, 07 Oct 2009, Rafael Garcia-Suarez wrote​:

The bug is most likely in the sort implementation, which is
notoriously thread-unsafe when it invokes sort subs.

In short, my advice would be : don't use sort() at all with a threaded perl.

Just to be sure​: That advice is in the context of using Safe.pm,
right? Or are there problems with sort() ourside safe compartments
under threading as well? Are there any open bugs about them?

Cheers,
-Jan

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 8, 2009

From @rgarcia

2009/10/8 Jan Dubois <jand@​activestate.com>​:

On Wed, 07 Oct 2009, Rafael Garcia-Suarez wrote​:

The bug is most likely in the sort implementation, which is
notoriously thread-unsafe when it invokes sort subs.

In short, my advice would be : don't use sort() at all with a threaded perl.

Just to be sure​: That advice is in the context of using Safe.pm,
right?  Or are there problems with sort() ourside safe compartments
under threading as well?  Are there any open bugs about them?

No, sort is thread-unsafe as long as the sort function is custom (and
not XS, IIRC). That's basically because it's called in a super-fast
way, for speed. I think there are bugs, but can't find one rapidly.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 8, 2009

From @demerphq

2009/10/8 Rafael Garcia-Suarez <rgs@​consttype.org>​:

2009/10/8 Jan Dubois <jand@​activestate.com>​:

On Wed, 07 Oct 2009, Rafael Garcia-Suarez wrote​:

The bug is most likely in the sort implementation, which is
notoriously thread-unsafe when it invokes sort subs.

In short, my advice would be : don't use sort() at all with a threaded perl.

Just to be sure​: That advice is in the context of using Safe.pm,
right?  Or are there problems with sort() ourside safe compartments
under threading as well?  Are there any open bugs about them?

No, sort is thread-unsafe as long as the sort function is custom (and
not XS, IIRC). That's basically because it's called in a super-fast
way, for speed. I think there are bugs, but can't find one rapidly.

Considering all Win32 builds pretty much are threaded id be surprised
if it was that easy. Having said that, maybe this explains some weird
win32 bugs...

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 8, 2009

From @csjewell

On Wed, 07 Oct 2009 10​:26 -0700, "David E. Wheeler"
<david@​kineticode.com> wrote​:

On Oct 7, 2009, at 10​:24 AM, Rafael Garcia-Suarez wrote​:

Even if we manage someone to fix threading, this will slow down
sort(). Threading also has a non-negligeable performance impact on
memory and speed of every perl program.

I think we should advice people in INSTALL to build with threading
only if they really need it, and tell OS vendors to disable threading
on their system perls.

Snow Leopard ships with a threaded Perl. Hell, I think Leopard did.
That cat might have left the bag.

And Strawberry is threaded. If I switched it to non-threaded now, I'd
be stood up to a wall and shot by the Padre people (they require
threading) among others.

--Curtis
--
Curtis Jewell
csjewell@​cpan.org http​://csjewell.dreamwidth.org/
perl@​csjewell.fastmail.us http​://csjewell.comyr.org/perl/

"Your random numbers are not that random" -- perl-5.10.1.tar.gz/util.c

Strawberry Perl for Windows betas​: http​://strawberryperl.com/beta/

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 8, 2009

From david@kineticode.com

On Oct 7, 2009, at 11​:25 PM, Rafael Garcia-Suarez wrote​:

Just to be sure​: That advice is in the context of using Safe.pm,
right? Or are there problems with sort() ourside safe compartments
under threading as well? Are there any open bugs about them?

No, sort is thread-unsafe as long as the sort function is custom (and
not XS, IIRC). That's basically because it's called in a super-fast
way, for speed. I think there are bugs, but can't find one rapidly.

It seems to me that such bugs should be fixed as they're identified.
Even if sorting becomes slower with threads, I'll take correct over
fast any day.

Best,

David

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 8, 2009

From @rgarcia

2009/10/8 David E. Wheeler <david@​kineticode.com>​:

On Oct 7, 2009, at 11​:25 PM, Rafael Garcia-Suarez wrote​:

Just to be sure​: That advice is in the context of using Safe.pm,
right?  Or are there problems with sort() ourside safe compartments
under threading as well?  Are there any open bugs about them?

No, sort is thread-unsafe as long as the sort function is custom (and
not XS, IIRC). That's basically because it's called in a super-fast
way, for speed. I think there are bugs, but can't find one rapidly.

It seems to me that such bugs should be fixed as they're identified. Even if
sorting becomes slower with threads, I'll take correct over fast any day.

I wholeheartedly agree. Someone should fix that. It's just not my
itch​: I don't build or use threaded perls.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 8, 2009

From @Tux

On Thu, 8 Oct 2009 09​:41​:42 -0700, "David E. Wheeler"
<david@​kineticode.com> wrote​:

On Oct 7, 2009, at 11​:25 PM, Rafael Garcia-Suarez wrote​:

Just to be sure​: That advice is in the context of using Safe.pm,
right? Or are there problems with sort() ourside safe compartments
under threading as well? Are there any open bugs about them?

No, sort is thread-unsafe as long as the sort function is custom (and
not XS, IIRC). That's basically because it's called in a super-fast
way, for speed. I think there are bugs, but can't find one rapidly.

It seems to me that such bugs should be fixed as they're identified.
Even if sorting becomes slower with threads, I'll take correct over
fast any day.

Then use an unthreaded perl, and you've got correct behaviour :)

As long as it doesn't slow down unthreaded builds, I'm fine with any
solution. I have never ever needed a threaded build for anything, and
speed is the major reason to build unthreaded perl from the beginning.

IIRC the worst case difference was up to 40% loss (gcc + threads vs
native cc no-threads), where threads was the biggest problem.

--
H.Merijn Brand http​://tux.nl Perl Monger http​://amsterdam.pm.org/
using & porting perl 5.6.2, 5.8.x, 5.10.x, 5.11.x on HP-UX 10.20, 11.00,
11.11, 11.23, and 11.31, OpenSuSE 10.3, 11.0, and 11.1, AIX 5.2 and 5.3.
http​://mirrors.develooper.com/hpux/ http​://www.test-smoke.org/
http​://qa.perl.org http​://www.goldmark.org/jeff/stupid-disclaimers/

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 8, 2009

From @demerphq

2009/10/8 Curtis Jewell <lists.perl.perl5-porters@​csjewell.fastmail.us>​:

On Wed, 07 Oct 2009 10​:26 -0700, "David E. Wheeler"
<david@​kineticode.com> wrote​:

On Oct 7, 2009, at 10​:24 AM, Rafael Garcia-Suarez wrote​:

Even if we manage someone to fix threading, this will slow down
sort(). Threading also has a non-negligeable performance impact on
memory and speed of every perl program.

I think we should advice people in INSTALL to build with threading
only if they really need it, and tell OS vendors to disable threading
on their system perls.

Snow Leopard ships with a threaded Perl. Hell, I think Leopard did.
That cat might have left the bag.

And Strawberry is threaded.  If I switched it to non-threaded now, I'd
be stood up to a wall and shot by the Padre people (they require
threading) among others.

Im inclined to say that a non-threaded win32 build is pretty close to broken.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 9, 2009

From badalex@gmail.com

On Wed Oct 07 08​:22​:57 2009, rgs@​consttype.org wrote​:

I don't think that *a and *b should be added to the default sharelist
of Safe. Actually the only glob there is *_, and I think it should
remain as is.

Hrm... Ok How does the below look to you?

Safe.pm | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)

Fix ->reval('sub { sort { "$a" <=> "$b" } }')->() on usethreads

This is broken on usethreads builds because we only store the text of
the stash/package (i.e. main) in COP's. When the anon sub executes, it
then uses the stash name to look stuff up using the current (and wrong)
PL_defstash/PL_curstash. The non threaded case stores references so its
not an issue.

To fix this modify ->reval() to detect if we are returning an anon sub.
If so return an anon sub which calls _safe_call_sv(..., $real_anon_sub)
to setup the right PL_defstash and friends.

--

--- /usr/lib/perl5/site_perl/5.10.1/Safe.pm 2009-08-25
01​:41​:11.000000000 -0600
+++ 5.10.1/Safe.pm 2009-10-08 23​:07​:33.027744998 -0600
@​@​ -2,6 +2,7 @​@​

use 5.003_11;
use strict;
+use Config qw(%Config);

$Safe​::VERSION = "2.19";

@​@​ -289,7 +290,13 @​@​
  my $root = $obj->{Root};

  my $evalsub = lexless_anon_sub($root,$strict, $expr);
- return Opcode​::_safe_call_sv($root, $obj->{Mask}, $evalsub);
+ my $ret = Opcode​::_safe_call_sv($root, $obj->{Mask}, $evalsub);
+
+ if ($Config{usethreads} && ref $ret eq 'CODE') {
+ return sub { Opcode​::_safe_call_sv($root, $obj->{Mask}, $ret) };
+ }
+
+ return $ret;
}

sub rdo {

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 9, 2009

From @steve-m-hay

demerphq wrote on 2009-10-08​:

2009/10/8 Curtis Jewell <lists.perl.perl5-porters@​csjewell.fastmail.us>​:

On Wed, 07 Oct 2009 10​:26 -0700, "David E. Wheeler"
<david@​kineticode.com> wrote​:

On Oct 7, 2009, at 10​:24 AM, Rafael Garcia-Suarez wrote​:

Even if we manage someone to fix threading, this will slow down
sort(). Threading also has a non-negligeable performance impact on
memory and speed of every perl program.

I think we should advice people in INSTALL to build with threading
only if they really need it, and tell OS vendors to disable threading
on their system perls.

Snow Leopard ships with a threaded Perl. Hell, I think Leopard did.
That cat might have left the bag.
And Strawberry is threaded.  If I switched it to non-threaded now, I'd
be stood up to a wall and shot by the Padre people (they require
threading) among others.
Im inclined to say that a non-threaded win32 build is pretty close to
broken.

Why do you say that? I always used to use a non-threaded perl on Win32, and only switched to threaded when I had to because mod_perl-2 needs it. Even now I still don't enable PERL_IMPLICIT_SYS because I have no need for the fork() emulation, and prefer Perl's malloc() (which can't be enabled with PERL_IMPLICIT_SYS) to the CRT's version.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 9, 2009

From @demerphq

2009/10/9 Steve Hay <SteveHay@​planit.com>​:

demerphq wrote on 2009-10-08​:

2009/10/8 Curtis Jewell <lists.perl.perl5-porters@​csjewell.fastmail.us>​:

On Wed, 07 Oct 2009 10​:26 -0700, "David E. Wheeler"
<david@​kineticode.com> wrote​:

On Oct 7, 2009, at 10​:24 AM, Rafael Garcia-Suarez wrote​:

Even if we manage someone to fix threading, this will slow down
sort(). Threading also has a non-negligeable performance impact on
memory and speed of every perl program.

I think we should advice people in INSTALL to build with threading
only if they really need it, and tell OS vendors to disable threading
on their system perls.

Snow Leopard ships with a threaded Perl. Hell, I think Leopard did.
That cat might have left the bag.
 And Strawberry is threaded.  If I switched it to non-threaded now, I'd
be stood up to a wall and shot by the Padre people (they require
threading) among others.
 Im inclined to say that a non-threaded win32 build is pretty close to
broken.

Why do you say that? I always used to use a non-threaded perl on Win32, and only switched to threaded when I had to because mod_perl-2 needs it. Even now I still don't enable PERL_IMPLICIT_SYS because I have no need for the fork() emulation, and prefer Perl's malloc() (which can't be enabled with PERL_IMPLICIT_SYS) to the CRT's version.

Ah well, then its just me. Anytime i used an unthreaded perl on
windows it was just pain.

Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 9, 2009

From jpl@research.att.com

David E. Wheeler noted​:

On Oct 7, 2009, at 11​:25 PM, Rafael Garcia-Suarez wrote​:

Just to be sure​: That advice is in the context of using Safe.pm,
right? Or are there problems with sort() ourside safe compartments
under threading as well? Are there any open bugs about them?

No, sort is thread-unsafe as long as the sort function is custom (and
not XS, IIRC). That's basically because it's called in a super-fast
way, for speed. I think there are bugs, but can't find one rapidly.

It seems to me that such bugs should be fixed as they're identified.
Even if sorting becomes slower with threads, I'll take correct over
fast any day.

I have never used threads, and I don't understand the details of
the problem caused by custom comparison routines, but I have
never let ignorance stand in the way of an opinion. Might it be
possible to put a global shared lock on the use of sort, so only
one thread at a time could be doing a sort? -- jpl

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 9, 2009

From @rgarcia

2009/10/9 John P. Linderman <jpl@​research.att.com>​:

David E. Wheeler noted​:

On Oct 7, 2009, at 11​:25 PM, Rafael Garcia-Suarez wrote​:

Just to be sure​: That advice is in the context of using Safe.pm,
right?  Or are there problems with sort() ourside safe compartments
under threading as well?  Are there any open bugs about them?

No, sort is thread-unsafe as long as the sort function is custom (and
not XS, IIRC). That's basically because it's called in a super-fast
way, for speed. I think there are bugs, but can't find one rapidly.

It seems to me that such bugs should be fixed as they're identified.
Even if sorting becomes slower with threads, I'll take correct over
fast any day.

I have never used threads, and I don't understand the details of
the problem caused by custom comparison routines, but I have
never let ignorance stand in the way of an opinion.  Might it be
possible to put a global shared lock on the use of sort, so only
one thread at a time could be doing a sort?  -- jpl

That way you will deadlock if your sort subroutine wants to wait for
another thread. (OK, that's probably very unlikely).
IIRC, the problem comes from the fact that pp_sort modifies the
optree, which is shared among threads.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 11, 2009

From @iabyn

On Fri, Oct 09, 2009 at 12​:37​:15PM +0200, Rafael Garcia-Suarez wrote​:

That way you will deadlock if your sort subroutine wants to wait for
another thread. (OK, that's probably very unlikely).
IIRC, the problem comes from the fact that pp_sort modifies the
optree, which is shared among threads.

Are you sure that's still the case? I thought that had been fixed ages
ago?

--
Dave's first rule of Opera​:
If something needs saying, say it​: don't warble it.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Oct 30, 2009

From @timbunce

On Sun, Oct 11, 2009 at 01​:35​:40PM +0100, Dave Mitchell wrote​:

On Fri, Oct 09, 2009 at 12​:37​:15PM +0200, Rafael Garcia-Suarez wrote​:

That way you will deadlock if your sort subroutine wants to wait for
another thread. (OK, that's probably very unlikely).
IIRC, the problem comes from the fact that pp_sort modifies the
optree, which is shared among threads.

Are you sure that's still the case? I thought that had been fixed ages
ago?

Specifically in http​://rt.perl.org/rt3/Public/Bug/Display.html?id=30333
I presume.

Tim.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 6, 2009

From @timbunce

On Tue, Oct 06, 2009 at 12​:55​:47PM -0700, Tim Bunce via RT wrote​:

Any news of progress with this bug?

Meanwhile, I have a workaround​: explicitly share the globs for a and b. In other words, add *a
and * b to the arguments of the share() method​:

$safe->share(qw(&debug *a *b));

To summarize the thread so far...
- it's confirmed that the bug is still present in 5.10.1.
- we digressed into a "don't use threaded perl" discussion
  but that's not an option for many people these days.
- some workarounds are possible, but *none work* for calls to
  sort() with custom sort subs in third-party code
  in other packages e.g., CPAN modules.

Tim.

p.s. It was suggested that sort with custom sort subs isn't thread safe
anyway. If that's true then it's a very serious problem. It's covered
by RT#30333 (http​://rt.perl.org/rt3/Public/Bug/Display.html?id=30333)
which is still marked Open. Yet I see tests for sort with a custom sub
in op/threads.t. Should RT#30333 be marked Closed?

If so, does that also fix RT#37508?
http​://rt.perl.org/rt3/Public/Bug/Display.html?id=37508

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 6, 2009

From @ikegami

On Fri, Nov 6, 2009 at 5​:43 AM, Tim Bunce <Tim.Bunce@​pobox.com> wrote​:

p.s. It was suggested that sort with custom sort subs isn't thread safe
anyway. If that's true then it's a very serious problem. It's covered
by RT#30333 (http​://rt.perl.org/rt3/Public/Bug/Display.html?id=30333)
which is still marked Open. Yet I see tests for sort with a custom sub
in op/threads.t. Should RT#30333 be marked Closed?

I can't reproduce it with new versions of Perl.
The caused identified in RT#30333 was solved by
http​://perl5.git.perl.org/perl.git/commit/9850bf21fc4ed69d8ddb0293df59411f891c62df

If so, does that also fix RT#37508?

http​://rt.perl.org/rt3/Public/Bug/Display.html?id=37508

I can't reproduce it with new versions of Perl.
The caused identified in RT#37508 was the same one identified in RT#30333.

I think both should be closed.
- Eric

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 7, 2009

From badalex@gmail.com

[ Resend I replied but RT does not show it yet +20 hours and neither
does any perl5-porters list i can see ]

On Fri, Nov 6, 2009 at 03​:43, Tim Bunce <Tim.Bunce@​pobox.com> wrote​:

On Tue, Oct 06, 2009 at 12​:55​:47PM -0700, Tim Bunce via RT wrote​:

To summarize the thread so far...
- it's confirmed that the bug is still present in 5.10.1.
- we digressed into a "don't use threaded perl" discussion
   but that's not an option for many people these days.

Yes, and thanks for pointing out that the above really has nothing to
do with *this* bug. Its not running a sort with threads its just
broken with a threaded *build* of perl. :)

- some workarounds are possible, but *none work* for calls to
sort() with custom sort subs in third-party code

Um the last patch I posted should...
http​://rt.perl.org/rt3/Ticket/Display.html?id=60374#txn-612980. And
while I found it quite ugly and im sure there are better ways to fix
this... (namely we could fix Opcode​::_safe_call_sv to do something
similar, or fix it as diagnosed in
http​://rt.perl.org/rt3/Ticket/Display.html?id=60374#txn-491794) I was
more or less looking for some kind of validation as to the approach
:).

But it seems RT never sent it anywhere... Mea culpa

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 7, 2009

From badalex@gmail.com

On Sat, Nov 7, 2009 at 10​:57, Alex Hunsaker <badalex@​gmail.com> wrote​:

Um the last patch I posted should...
http​://rt.perl.org/rt3/Ticket/Display.html?id=60374#txn-612980.  And
while I found it quite ugly

For reference here is said patch also find attached for the inevitable
gmail white space damage.

Subject​: Fix ->reval('sub { sort { "$a" <=> "$b" } }')->() on usethreads

This is broken on usethreads builds because we only store the text of
the stash/package (i.e. main) in COP's. When the anon sub executes, it
then uses the stash name to look stuff up using the current (and wrong)
PL_defstash/PL_curstash. The non threaded case stores references so its
not an issue.

To fix this modify ->reval() to detect if we are returning an anon sub.
If so return an anon sub which calls _safe_call_sv(..., $real_anon_sub)
to setup the right PL_defstash and friends.

--

Safe.pm | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)

Inline Patch
diff --git a/ext/Safe/Safe.pm b/ext/Safe/Safe.pm
index ff099ec..e9c5dfa 100644
--- a/ext/Safe/Safe.pm
+++ b/ext/Safe/Safe.pm
@@ -2,6 +2,7 @@ package Safe;

 use 5.003_11;
 use strict;
+use Config qw(%Config);

 $Safe::VERSION = "2.18";

@@ -289,7 +290,13 @@ sub reval {
     my $root = $obj->{Root};

     my $evalsub = lexless_anon_sub($root,$strict, $expr);
-    return Opcode::_safe_call_sv($root, $obj->{Mask}, $evalsub);
+    my $ret = Opcode::_safe_call_sv($root, $obj->{Mask}, $evalsub);
+
+    if ($Config{usethreads} && ref $ret eq 'CODE') {
+       return sub { Opcode::_safe_call_sv($root, $obj->{Mask}, $ret) };
+    }
+
+    return $ret;
 }

 sub rdo {
@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 7, 2009

From badalex@gmail.com

safe_sort_threads.patch
Subject: Fix ->reval('sub { sort { "$a" <=> "$b" } }')->() on usethreads

This is broken on usethreads builds because we only store the text of
the stash/package (i.e. main) in COP's. When the anon sub executes, it
then uses the stash name to look stuff up using the current (and wrong)
PL_defstash/PL_curstash. The non threaded case stores references so its
not an issue.

To fix this modify ->reval() to detect if we are returning an anon sub.
If so return an anon sub which calls _safe_call_sv(..., $real_anon_sub)
to setup the right PL_defstash and friends.
--

Safe.pm | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/ext/Safe/Safe.pm b/ext/Safe/Safe.pm
index ff099ec..e9c5dfa 100644
--- a/ext/Safe/Safe.pm
+++ b/ext/Safe/Safe.pm
@@ -2,6 +2,7 @@ package Safe;
 
 use 5.003_11;
 use strict;
+use Config qw(%Config);
 
 $Safe::VERSION = "2.18";
 
@@ -289,7 +290,13 @@ sub reval {
     my $root = $obj->{Root};
 
     my $evalsub = lexless_anon_sub($root,$strict, $expr);
-    return Opcode::_safe_call_sv($root, $obj->{Mask}, $evalsub);
+    my $ret = Opcode::_safe_call_sv($root, $obj->{Mask}, $evalsub);
+
+    if ($Config{usethreads} && ref $ret eq 'CODE') {
+       return sub { Opcode::_safe_call_sv($root, $obj->{Mask}, $ret) };
+    }
+
+    return $ret;
 }
 
 sub rdo {
@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 7, 2009

From ben@morrow.me.uk

Quoth badalex@​gmail.com (Alex Hunsaker)​:

 my $evalsub = lexless\_anon\_sub\($root\,$strict\, $expr\);

- return Opcode​::_safe_call_sv($root, $obj->{Mask}, $evalsub);
+ my $ret = Opcode​::_safe_call_sv($root, $obj->{Mask}, $evalsub);
+
+ if ($Config{usethreads} && ref $ret eq 'CODE') {

This should use Scalar​::Util​::reftype, in case the coderef is blessed.

Ben

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 7, 2009

From badalex@gmail.com

On Fri, Nov 6, 2009 at 03​:44, Tim Bunce via RT
<perlbug-followup@​perl.org> wrote​:

On Tue, Oct 06, 2009 at 12​:55​:47PM -0700, Tim Bunce via RT wrote​:

Any news of progress with this bug?

To summarize the thread so far...
- it's confirmed that the bug is still present in 5.10.1.
- we digressed into a "don't use threaded perl" discussion
   but that's not an option for many people these days.

Yes, and thanks for pointing out that the above really has nothing to
do with *this* bug. Its not running a sort with threads its just
broken with a threaded *build* of perl. :)

- some workarounds are possible, but *none work* for calls to
   sort() with custom sort subs in third-party code

Um the last patch I posted should...
http​://rt.perl.org/rt3/Ticket/Display.html?id=60374#txn-612980. And
while I found it quite ugly and im sure there are better ways to fix
this... (namely we could fix Opcode​::_safe_call_sv to do something
similar, or fix it as diagnosed in
http​://rt.perl.org/rt3/Ticket/Display.html?id=60374#txn-491794) I was
more or less looking for some kind of validation as to the approach
:).

People seemed to prefer to talk about how you should never be using a
threaded perl anyways... *shrug* Perhaps I should submit a patch to
disable -DUSETHREADS instead ?

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 8, 2009

From @timbunce

On Fri, Nov 06, 2009 at 02​:30​:47PM -0700, Alex Hunsaker wrote​:

Um the last patch I posted should...
http​://rt.perl.org/rt3/Ticket/Display.html?id=60374#txn-612980. And
while I found it quite ugly and im sure there are better ways to fix
this... (namely we could fix Opcode​::_safe_call_sv to do something
similar, or fix it as diagnosed in
http​://rt.perl.org/rt3/Ticket/Display.html?id=60374#txn-491794)
I was more or less looking for some kind of validation as to the
approach :).

I don't think the "detect if we are returning an anon sub" approach will
work because the reval() may have compiled some named subs that will be
invoked later. That's the case with PostgrSQL's embedded plperl, for example.

http​://rt.perl.org/rt3/Public/Bug/Display.html?id=60374#txn-491794 says
  "pp_sort uses CopSTASH(PL_curcop) to get the package stash (symbol
  table) in order to locate the $a and $b variables in it."

While pp_sort does call CopSTASH(PL_curcop) directly, all it does with
it is store the value in PL_sortstash which, as far as I can tell, isn't
used or documented anywhere at all. (It could possibly be deleted.)

The code to lookup the $a and $b uses​:
  PL_firstgv = gv_fetchpvs("a", GV_ADD|GV_NOTQUAL, SVt_PV);
  PL_secondgv = gv_fetchpvs("b", GV_ADD|GV_NOTQUAL, SVt_PV);
and it's Perl_gv_fetchpvn_flags that ends up calling CopSTASH(PL_curcop).

If there was a way to get the stash pointer (not name) that was in
effect at the time the code was compiled we could use that to lookup $a
and $b.

There are stash pointers, er, stashed in lots of places.
One approach might be CvSTASH(find_runcv(...))

Tim.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 9, 2009

From badalex@gmail.com

On Sun, Nov 8, 2009 at 16​:36, Tim Bunce <Tim.Bunce@​pobox.com> wrote​:

I don't think the "detect if we are returning an anon sub" approach will
work because the reval() may have compiled some named subs that will be
invoked later. That's the case with PostgrSQL's embedded plperl, for example.

BTW I just tested and it does fix PostgreSQL

create or replace function trustedsort()
returns int
as $$
my @​arr = (5, 4, 3, 2, 1);
my @​sorted = sort { elog(NOTICE, "$a $b"); $a <=> $b } @​arr;
return 1;
$$
language 'plperl';

unpatched Safe.pm​:
select trustedsort();
NOTICE​:
CONTEXT​: PL/Perl function "trustedsort"
NOTICE​:
CONTEXT​: PL/Perl function "trustedsort"
NOTICE​:
CONTEXT​: PL/Perl function "trustedsort"
NOTICE​:
CONTEXT​: PL/Perl function "trustedsort"
NOTICE​:
CONTEXT​: PL/Perl function "trustedsort"
NOTICE​:
CONTEXT​: PL/Perl function "trustedsort"
NOTICE​:
CONTEXT​: PL/Perl function "trustedsort"
NOTICE​:
CONTEXT​: PL/Perl function "trustedsort"

trustedsort


  1
(1 row)

w patched​:
select trustedsort();
NOTICE​: 5 4
CONTEXT​: PL/Perl function "trustedsort"
NOTICE​: 3 2
CONTEXT​: PL/Perl function "trustedsort"
NOTICE​: 4 2
CONTEXT​: PL/Perl function "trustedsort"
NOTICE​: 4 3
CONTEXT​: PL/Perl function "trustedsort"
NOTICE​: 2 1
CONTEXT​: PL/Perl function "trustedsort"
trustedsort


  1

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 9, 2009

From @timbunce

On Sun, Nov 08, 2009 at 05​:56​:21PM -0700, Alex Hunsaker wrote​:

On Sun, Nov 8, 2009 at 16​:36, Tim Bunce <Tim.Bunce@​pobox.com> wrote​:

I don't think the "detect if we are returning an anon sub" approach will
work because the reval() may have compiled some named subs that will be
invoked later. That's the case with PostgrSQL's embedded plperl, for example.

BTW I just tested and it does fix PostgreSQL

Ah, yes, PostgreSQL's embedded plperl does compile anon-subs. Thanks.
Also your approach does seem to fix named subs invoked via the anon sub.
It's unfortunate that the cost is two extra sub calls to invoke the
anon sub :(

I've attached a patch to Safe, based on your idea, slightly optimized
and generalized, and including some tests.

I've tested on a bunch of perl's I have lying around​:
$ foreachperl makefreshtest
[...]
  ok​: /usr/local/perl511-i/bin
  ok​: /usr/local/perl510-pure/bin
  ok​: /usr/local/perl510-thr64/bin
  ok​: /usr/local/perl510/bin
  ok​: /usr/local/perl58-i/bin
  ok​: /usr/local/perl588-64/bin
  ok​: /usr/local/perl588-thr/bin
  ok​: /usr/local/perl589-i/bin
  ok​: /usr/local/perl589/bin

Thanks Alex!

Tim.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 9, 2009

From @timbunce

Safe-2.19-perl-rt-60374.patch
diff -rNu Safe-2.19-orig/MANIFEST Safe-2.19/MANIFEST
--- Safe-2.19-orig/MANIFEST	2009-06-28 15:14:49.000000000 +0100
+++ Safe-2.19/MANIFEST	2009-11-09 15:56:28.000000000 +0000
@@ -8,5 +8,6 @@
 t/safe3.t
 t/safeload.t
 t/safeops.t
+t/safesort.t
 t/safeuniversal.t
 META.yml                                 Module meta-data (added by MakeMaker)
diff -rNu Safe-2.19-orig/Safe.pm Safe-2.19/Safe.pm
--- Safe-2.19-orig/Safe.pm	2009-08-25 08:41:11.000000000 +0100
+++ Safe-2.19/Safe.pm	2009-11-09 16:10:47.000000000 +0000
@@ -2,6 +2,9 @@
 
 use 5.003_11;
 use strict;
+use Scalar::Util qw(reftype);
+use Config qw(%Config);
+use constant is_usethreads => $Config{usethreads};
 
 $Safe::VERSION = "2.19";
 
@@ -288,8 +291,26 @@
     my ($obj, $expr, $strict) = @_;
     my $root = $obj->{Root};
 
-    my $evalsub = lexless_anon_sub($root,$strict, $expr);
-    return Opcode::_safe_call_sv($root, $obj->{Mask}, $evalsub);
+    my $evalsub = lexless_anon_sub($root, $strict, $expr);
+    my @ret = (wantarray)
+        ?        Opcode::_safe_call_sv($root, $obj->{Mask}, $evalsub)
+        : scalar Opcode::_safe_call_sv($root, $obj->{Mask}, $evalsub);
+
+    # RT#60374: Safe.pm sort {} bug with -Dusethreads
+    # If the Safe eval returns a code ref in a perl compiled with usethreads
+    # then wrap code ref with _safe_call_sv so that, when called, the
+    # execution will happen with the compartment fully 'in effect'.
+    # Needed to fix sort blocks that reference $a & $b and
+    # possibly other subtle issues.
+    if (is_usethreads()) {
+        for my $ret (@ret) { # edit (via alias) any CODE refs
+            next unless (reftype($ret)||'') eq 'CODE';
+            my $sub = $ret; # avoid closure problems
+            $ret = sub { Opcode::_safe_call_sv($root, $obj->{Mask}, $sub) };
+        }
+    }
+
+    return (wantarray) ? @ret : $ret[0];
 }
 
 sub rdo {
diff -rNu Safe-2.19-orig/t/safesort.t Safe-2.19/t/safesort.t
--- Safe-2.19-orig/t/safesort.t	1970-01-01 01:00:00.000000000 +0100
+++ Safe-2.19/t/safesort.t	2009-11-09 15:53:10.000000000 +0000
@@ -0,0 +1,37 @@
+#!./perl -w
+$|=1;
+BEGIN {
+    if($ENV{PERL_CORE}) {
+	chdir 't' if -d 't';
+	@INC = '../lib';
+    } 
+    require Config; import Config;
+    if ($Config{'extensions'} !~ /\bOpcode\b/ && $Config{'osname'} ne 'VMS') {
+        print "1..0\n";
+        exit 0;
+    }
+}
+
+use Safe 1.00;
+use Test::More tests => 4;
+
+my $safe = Safe->new('PLPerl');
+$safe->permit_only(qw(:default sort));
+
+my $func = $safe->reval(<<'EOS');
+
+    # uses quotes in { "$a" <=> $b } to avoid the optimizer replacing the block
+    # with a hardwired comparison
+    { package Pkg; sub p_sort { return sort { "$a" <=> $b } qw(2 1 3); } }
+                   sub l_sort { return sort { "$a" <=> $b } qw(2 1 3); }
+
+    return sub { return join(",",l_sort()), join(",",Pkg::p_sort()) }
+
+EOS
+
+is $@, '', 'reval should not fail';
+is ref $func, 'CODE', 'reval should return a CODE ref';
+
+my ($l_sorted, $p_sorted) = $func->();
+is $l_sorted, "1,2,3";
+is $p_sorted, "1,2,3";
@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 9, 2009

From david@kineticode.com

On Nov 9, 2009, at 8​:21 AM, Tim Bunce wrote​:

Ah, yes, PostgreSQL's embedded plperl does compile anon-subs. Thanks.
Also your approach does seem to fix named subs invoked via the anon
sub.
It's unfortunate that the cost is two extra sub calls to invoke the
anon sub :(

I've attached a patch to Safe, based on your idea, slightly optimized
and generalized, and including some tests.

Perhaps it got lost in the shuffle, and I'm certainly glad to see a
Safe.pm issue addressed, but is there still an existing issue with
sort and -Dusethreads?

Best,

David

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 9, 2009

From @timbunce

On Mon, Nov 09, 2009 at 09​:10​:18AM -0800, David E. Wheeler wrote​:

On Nov 9, 2009, at 8​:21 AM, Tim Bunce wrote​:

Ah, yes, PostgreSQL's embedded plperl does compile anon-subs. Thanks.
Also your approach does seem to fix named subs invoked via the anon sub.
It's unfortunate that the cost is two extra sub calls to invoke the
anon sub :(

I've attached a patch to Safe, based on your idea, slightly optimized
and generalized, and including some tests.

Perhaps it got lost in the shuffle,

It reached the ticket ok​:
http​://rt.perl.org/rt3/Ticket/Display.html?id=60374#txn-628757

and I'm certainly glad to see a Safe.pm issue addressed, but is there
still an existing issue with sort and -Dusethreads?

In http​://rt.perl.org/rt3/Ticket/Display.html?id=60374#txn-627795
Eric Brine says he can't reproduce them, so
  http​://rt.perl.org/rt3/Public/Bug/Display.html?id=37508
  http​://rt.perl.org/rt3/Public/Bug/Display.html?id=30333
should be closed. (I don't know who gets to do that.)

Tim.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 10, 2009

From @nwc10

On Sun, Nov 08, 2009 at 11​:36​:46PM +0000, Tim Bunce wrote​:

On Fri, Nov 06, 2009 at 02​:30​:47PM -0700, Alex Hunsaker wrote​:

Um the last patch I posted should...
http​://rt.perl.org/rt3/Ticket/Display.html?id=60374#txn-612980. And
while I found it quite ugly and im sure there are better ways to fix
this... (namely we could fix Opcode​::_safe_call_sv to do something
similar, or fix it as diagnosed in
http​://rt.perl.org/rt3/Ticket/Display.html?id=60374#txn-491794)
I was more or less looking for some kind of validation as to the
approach :).

I don't think the "detect if we are returning an anon sub" approach will
work because the reval() may have compiled some named subs that will be
invoked later. That's the case with PostgrSQL's embedded plperl, for example.

http​://rt.perl.org/rt3/Public/Bug/Display.html?id=60374#txn-491794 says
"pp_sort uses CopSTASH(PL_curcop) to get the package stash (symbol
table) in order to locate the $a and $b variables in it."

While pp_sort does call CopSTASH(PL_curcop) directly, all it does with
it is store the value in PL_sortstash which, as far as I can tell, isn't
used or documented anywhere at all. (It could possibly be deleted.)

The code to lookup the $a and $b uses​:
PL_firstgv = gv_fetchpvs("a", GV_ADD|GV_NOTQUAL, SVt_PV);
PL_secondgv = gv_fetchpvs("b", GV_ADD|GV_NOTQUAL, SVt_PV);
and it's Perl_gv_fetchpvn_flags that ends up calling CopSTASH(PL_curcop).

If there was a way to get the stash pointer (not name) that was in
effect at the time the code was compiled we could use that to lookup $a
and $b.

The logical way to do this would be to indirect via an entry in the
subroutine's pad, as the reference would be correctly translated by the
ithreads clone code. This probably isn't *trivial* to implement, as a
subroutine can contain code compiled in more than one package, but I think*
it would be viable at compile time to loop up/store if not there yet, the
stash pointer, under the stash's name.

There are stash pointers, er, stashed in lots of places.
One approach might be CvSTASH(find_runcv(...))

I think that that doesn't cope with

package foo;
sub bar {
  # here package is foo
  package baz;
  # here package is baz
}

Nicholas Clark

* I think. I don't know this stuff well.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 10, 2009

From @timbunce

On Tue, Nov 10, 2009 at 02​:56​:36PM +0000, Nicholas Clark wrote​:

http​://rt.perl.org/rt3/Public/Bug/Display.html?id=60374#txn-491794 says
"pp_sort uses CopSTASH(PL_curcop) to get the package stash (symbol
table) in order to locate the $a and $b variables in it."

While pp_sort does call CopSTASH(PL_curcop) directly, all it does with
it is store the value in PL_sortstash which, as far as I can tell, isn't
used or documented anywhere at all. (It could possibly be deleted.)

The code to lookup the $a and $b uses​:
PL_firstgv = gv_fetchpvs("a", GV_ADD|GV_NOTQUAL, SVt_PV);
PL_secondgv = gv_fetchpvs("b", GV_ADD|GV_NOTQUAL, SVt_PV);
and it's Perl_gv_fetchpvn_flags that ends up calling CopSTASH(PL_curcop).

If there was a way to get the stash pointer (not name) that was in
effect at the time the code was compiled we could use that to lookup $a
and $b.

The logical way to do this would be to indirect via an entry in the
subroutine's pad, as the reference would be correctly translated by the
ithreads clone code. This probably isn't *trivial* to implement, as a
subroutine can contain code compiled in more than one package, but I think*
it would be viable at compile time to loop up/store if not there yet, the
stash pointer, under the stash's name.

There are stash pointers, er, stashed in lots of places.
One approach might be CvSTASH(find_runcv(...))

I think that that doesn't cope with

package foo;
sub bar {
# here package is foo
package baz;
# here package is baz
}

I couldn't get it to work even without that (but trying to follow the
inside-safe vs outside-safe and compile-time vs run-time intricacies
made my rusty old head spin, so I could have done something silly).

Alex's approach, of arranging to execute the closure 'inside' the
compartment, seems like a good idea from a 'safety' point of view
anyway. The fact it works around the bug is a bonus.

On the other hand, the fact it costs two extra sub calls is unfortunate.
I tried to look for a way to the PL_defstash switch into pp_entersub but
couldn't see a viable approach.

* I think. I don't know this stuff well.

Me neither. It's been many years since I helped develop Opcode and Safe.
(I recall refering to Safe as a "failed experiment" more than once.)

Tim.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 16, 2009

From @timbunce

I'm not sure who to ask to apply the patch to Safe.pm.
I've created a ticket on rt.cpan.org to ensure it doesn't slip though the cracks from that side​:
https://rt.cpan.org/Ticket/Display.html?id=51574

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 30, 2009

From @rgs

Now fixed with Safe 2.20.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 30, 2009

@rgs - Status changed from 'open' to 'resolved'

@p5pRT p5pRT closed this Nov 30, 2009
@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Feb 2, 2010

From @steve-m-hay

demerphq wrote on 2009-10-08​:

2009/10/8 Curtis Jewell <lists.perl.perl5-porters@​csjewell.fastmail.us>​:

On Wed, 07 Oct 2009 10​:26 -0700, "David E. Wheeler"
<david@​kineticode.com> wrote​:

On Oct 7, 2009, at 10​:24 AM, Rafael Garcia-Suarez wrote​:

Even if we manage someone to fix threading, this will slow down
sort(). Threading also has a non-negligeable performance impact on
memory and speed of every perl program.

I think we should advice people in INSTALL to build with threading
only if they really need it, and tell OS vendors to disable threading
on their system perls.

Snow Leopard ships with a threaded Perl. Hell, I think Leopard did.
That cat might have left the bag.
And Strawberry is threaded.  If I switched it to non-threaded now, I'd
be stood up to a wall and shot by the Padre people (they require
threading) among others.
Im inclined to say that a non-threaded win32 build is pretty close to
broken.

Why do you say that? I always used to use a non-threaded perl on Win32, and only switched to threaded when I had to because mod_perl-2 needs it. Even now I still don't enable PERL_IMPLICIT_SYS because I have no need for the fork() emulation, and prefer Perl's malloc() (which can't be enabled with PERL_IMPLICIT_SYS) to the CRT's version.

@p5pRT p5pRT added the Severity Low label Oct 18, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant
You can’t perform that action at this time.