-
Notifications
You must be signed in to change notification settings - Fork 550
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Safe.pm sort {} bug with -Dusethreads #9557
Comments
From badalex@gmail.comWhen using a threading perl and you reval an anon sub that contains a sort, (see attached perl.v for perl -V output) without -Dusethreads tsafe.pluse Safe; my $safe = Safe->new('PLPerl'); sub debug { my $func = $safe->reval(<<'z'); #my $f = eval 'package PLPerl; sub { @_=(); $func->(); }'; die |
From badalex@gmail.com |
From badalex@gmail.com |
From badalex@gmail.comAndrew Gierth <andrew@tao11.riddles.org.uk> managed to track it down If it helps any, I've tracked down in the perl guts exactly why this cop.h: struct cop { A COP in perl is a control operation, basically a compiled statement, Notice, though, that without ithreads, the COP points directly to the So with ithreads enabled, pp_sort looks up the package stash by name, So there are three factors involved: 1) the change in layout of COP with ithreads enabled However, I have no idea why Perl has this difference between threaded |
badalex@gmail.com - Status changed from 'new' to 'open' |
From badalex@gmail.comAndrew Gierth <andrew@tao11.riddles.org.uk> managed to track it down If it helps any, I've tracked down in the perl guts exactly why this cop.h: struct cop { A COP in perl is a control operation, basically a compiled statement, Notice, though, that without ithreads, the COP points directly to the So with ithreads enabled, pp_sort looks up the package stash by name, So there are three factors involved: 1) the change in layout of COP with ithreads enabled However, I have no idea why Perl has this difference between threaded |
From @timbunceAny news of progress with this bug? Meanwhile, I have a workaround: explicitly share the globs for a and b. In other words, add *a $safe->share(qw(&debug *a *b)); |
From badalex@gmail.comOn Tue, Oct 6, 2009 at 13:55, Tim Bunce via RT
Nope still broke for me on 5.10.1 Here are the workaround I know about: $func->(); 4) fix perl :) as mentioned in the previous message
Yeah as noted above that should work. Unless you ever sort inside a use Safe(); my $safe = Safe->new('PLPerl'); my $func = $safe->reval(<<'z'); $func->(); package PLPerl; $ perl tsafe.pl |
From @rgarcia2009/10/6 Tim Bunce via RT <perlbug-followup@perl.org>:
I don't think that *a and *b should be added to the default sharelist The bug is most likely in the sort implementation, which is In short, my advice would be : don't use sort() at all with a threaded perl. |
From david@kineticode.comOn Oct 7, 2009, at 8:22 AM, Rafael Garcia-Suarez wrote:
Say what? Sure, on current releases, but surely that should be fixed. David |
From badalex@gmail.comOn Wed Oct 07 08:22:57 2009, rgs@consttype.org wrote:
Um... you do realize its just a threaded *build* of perl. Nothing here |
From @rgarcia2009/10/7 David E. Wheeler <david@kineticode.com>:
Even if we manage someone to fix threading, this will slow down I think we should advice people in INSTALL to build with threading |
From david@kineticode.comOn Oct 7, 2009, at 10:24 AM, Rafael Garcia-Suarez wrote:
Snow Leopard ships with a threaded Perl. Hell, I think Leopard did. Best, David |
From @janduboisOn Wed, 07 Oct 2009, Rafael Garcia-Suarez wrote:
Just to be sure: That advice is in the context of using Safe.pm, Cheers, |
From @rgarcia2009/10/8 Jan Dubois <jand@activestate.com>:
No, sort is thread-unsafe as long as the sort function is custom (and |
From @demerphq2009/10/8 Rafael Garcia-Suarez <rgs@consttype.org>:
Considering all Win32 builds pretty much are threaded id be surprised Yves -- |
From @csjewellOn Wed, 07 Oct 2009 10:26 -0700, "David E. Wheeler"
And Strawberry is threaded. If I switched it to non-threaded now, I'd --Curtis "Your random numbers are not that random" -- perl-5.10.1.tar.gz/util.c Strawberry Perl for Windows betas: http://strawberryperl.com/beta/ |
From david@kineticode.comOn Oct 7, 2009, at 11:25 PM, Rafael Garcia-Suarez wrote:
It seems to me that such bugs should be fixed as they're identified. Best, David |
From @rgarcia2009/10/8 David E. Wheeler <david@kineticode.com>:
I wholeheartedly agree. Someone should fix that. It's just not my |
From @TuxOn Thu, 8 Oct 2009 09:41:42 -0700, "David E. Wheeler"
Then use an unthreaded perl, and you've got correct behaviour :) As long as it doesn't slow down unthreaded builds, I'm fine with any IIRC the worst case difference was up to 40% loss (gcc + threads vs -- |
From @demerphq2009/10/8 Curtis Jewell <lists.perl.perl5-porters@csjewell.fastmail.us>:
Im inclined to say that a non-threaded win32 build is pretty close to broken. Yves -- |
From badalex@gmail.comOn Wed Oct 07 08:22:57 2009, rgs@consttype.org wrote:
Hrm... Ok How does the below look to you? Safe.pm | 9 ++++++++- Fix ->reval('sub { sort { "$a" <=> "$b" } }')->() on usethreads This is broken on usethreads builds because we only store the text of To fix this modify ->reval() to detect if we are returning an anon sub. -- --- /usr/lib/perl5/site_perl/5.10.1/Safe.pm 2009-08-25 use 5.003_11; $Safe::VERSION = "2.19"; @@ -289,7 +290,13 @@ my $evalsub = lexless_anon_sub($root,$strict, $expr); sub rdo { |
From @steve-m-haydemerphq wrote on 2009-10-08:
Why do you say that? I always used to use a non-threaded perl on Win32, and only switched to threaded when I had to because mod_perl-2 needs it. Even now I still don't enable PERL_IMPLICIT_SYS because I have no need for the fork() emulation, and prefer Perl's malloc() (which can't be enabled with PERL_IMPLICIT_SYS) to the CRT's version. |
From @demerphq2009/10/9 Steve Hay <SteveHay@planit.com>:
Ah well, then its just me. Anytime i used an unthreaded perl on Yves -- |
From jpl@research.att.comDavid E. Wheeler noted:
I have never used threads, and I don't understand the details of |
From @rgarcia2009/10/9 John P. Linderman <jpl@research.att.com>:
That way you will deadlock if your sort subroutine wants to wait for |
From @iabynOn Fri, Oct 09, 2009 at 12:37:15PM +0200, Rafael Garcia-Suarez wrote:
Are you sure that's still the case? I thought that had been fixed ages -- |
From @timbunceOn Sun, Oct 11, 2009 at 01:35:40PM +0100, Dave Mitchell wrote:
Specifically in http://rt.perl.org/rt3/Public/Bug/Display.html?id=30333 Tim. |
From @timbunceOn Tue, Oct 06, 2009 at 12:55:47PM -0700, Tim Bunce via RT wrote:
To summarize the thread so far... Tim. p.s. It was suggested that sort with custom sort subs isn't thread safe If so, does that also fix RT#37508? |
From @ikegamiOn Fri, Nov 6, 2009 at 5:43 AM, Tim Bunce <Tim.Bunce@pobox.com> wrote:
I can't reproduce it with new versions of Perl. If so, does that also fix RT#37508?
I can't reproduce it with new versions of Perl. I think both should be closed. |
From badalex@gmail.com[ Resend I replied but RT does not show it yet +20 hours and neither On Fri, Nov 6, 2009 at 03:43, Tim Bunce <Tim.Bunce@pobox.com> wrote:
Yes, and thanks for pointing out that the above really has nothing to
Um the last patch I posted should... But it seems RT never sent it anywhere... Mea culpa |
From badalex@gmail.comOn Sat, Nov 7, 2009 at 10:57, Alex Hunsaker <badalex@gmail.com> wrote:
For reference here is said patch also find attached for the inevitable Subject: Fix ->reval('sub { sort { "$a" <=> "$b" } }')->() on usethreads This is broken on usethreads builds because we only store the text of To fix this modify ->reval() to detect if we are returning an anon sub. -- Safe.pm | 9 ++++++++- Inline Patchdiff --git a/ext/Safe/Safe.pm b/ext/Safe/Safe.pm
index ff099ec..e9c5dfa 100644
--- a/ext/Safe/Safe.pm
+++ b/ext/Safe/Safe.pm
@@ -2,6 +2,7 @@ package Safe;
use 5.003_11;
use strict;
+use Config qw(%Config);
$Safe::VERSION = "2.18";
@@ -289,7 +290,13 @@ sub reval {
my $root = $obj->{Root};
my $evalsub = lexless_anon_sub($root,$strict, $expr);
- return Opcode::_safe_call_sv($root, $obj->{Mask}, $evalsub);
+ my $ret = Opcode::_safe_call_sv($root, $obj->{Mask}, $evalsub);
+
+ if ($Config{usethreads} && ref $ret eq 'CODE') {
+ return sub { Opcode::_safe_call_sv($root, $obj->{Mask}, $ret) };
+ }
+
+ return $ret;
}
sub rdo { |
From badalex@gmail.comsafe_sort_threads.patchSubject: Fix ->reval('sub { sort { "$a" <=> "$b" } }')->() on usethreads
This is broken on usethreads builds because we only store the text of
the stash/package (i.e. main) in COP's. When the anon sub executes, it
then uses the stash name to look stuff up using the current (and wrong)
PL_defstash/PL_curstash. The non threaded case stores references so its
not an issue.
To fix this modify ->reval() to detect if we are returning an anon sub.
If so return an anon sub which calls _safe_call_sv(..., $real_anon_sub)
to setup the right PL_defstash and friends.
--
Safe.pm | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/ext/Safe/Safe.pm b/ext/Safe/Safe.pm
index ff099ec..e9c5dfa 100644
--- a/ext/Safe/Safe.pm
+++ b/ext/Safe/Safe.pm
@@ -2,6 +2,7 @@ package Safe;
use 5.003_11;
use strict;
+use Config qw(%Config);
$Safe::VERSION = "2.18";
@@ -289,7 +290,13 @@ sub reval {
my $root = $obj->{Root};
my $evalsub = lexless_anon_sub($root,$strict, $expr);
- return Opcode::_safe_call_sv($root, $obj->{Mask}, $evalsub);
+ my $ret = Opcode::_safe_call_sv($root, $obj->{Mask}, $evalsub);
+
+ if ($Config{usethreads} && ref $ret eq 'CODE') {
+ return sub { Opcode::_safe_call_sv($root, $obj->{Mask}, $ret) };
+ }
+
+ return $ret;
}
sub rdo {
|
From ben@morrow.me.ukQuoth badalex@gmail.com (Alex Hunsaker):
This should use Scalar::Util::reftype, in case the coderef is blessed. Ben |
From badalex@gmail.comOn Fri, Nov 6, 2009 at 03:44, Tim Bunce via RT
Yes, and thanks for pointing out that the above really has nothing to
Um the last patch I posted should... People seemed to prefer to talk about how you should never be using a |
From @timbunceOn Fri, Nov 06, 2009 at 02:30:47PM -0700, Alex Hunsaker wrote:
I don't think the "detect if we are returning an anon sub" approach will http://rt.perl.org/rt3/Public/Bug/Display.html?id=60374#txn-491794 says While pp_sort does call CopSTASH(PL_curcop) directly, all it does with The code to lookup the $a and $b uses: If there was a way to get the stash pointer (not name) that was in There are stash pointers, er, stashed in lots of places. Tim. |
From badalex@gmail.comOn Sun, Nov 8, 2009 at 16:36, Tim Bunce <Tim.Bunce@pobox.com> wrote:
BTW I just tested and it does fix PostgreSQL create or replace function trustedsort() unpatched Safe.pm: trustedsort 1 w patched: 1 |
From @timbunceOn Sun, Nov 08, 2009 at 05:56:21PM -0700, Alex Hunsaker wrote:
Ah, yes, PostgreSQL's embedded plperl does compile anon-subs. Thanks. I've attached a patch to Safe, based on your idea, slightly optimized I've tested on a bunch of perl's I have lying around: Thanks Alex! Tim. |
From @timbunceSafe-2.19-perl-rt-60374.patchdiff -rNu Safe-2.19-orig/MANIFEST Safe-2.19/MANIFEST
--- Safe-2.19-orig/MANIFEST 2009-06-28 15:14:49.000000000 +0100
+++ Safe-2.19/MANIFEST 2009-11-09 15:56:28.000000000 +0000
@@ -8,5 +8,6 @@
t/safe3.t
t/safeload.t
t/safeops.t
+t/safesort.t
t/safeuniversal.t
META.yml Module meta-data (added by MakeMaker)
diff -rNu Safe-2.19-orig/Safe.pm Safe-2.19/Safe.pm
--- Safe-2.19-orig/Safe.pm 2009-08-25 08:41:11.000000000 +0100
+++ Safe-2.19/Safe.pm 2009-11-09 16:10:47.000000000 +0000
@@ -2,6 +2,9 @@
use 5.003_11;
use strict;
+use Scalar::Util qw(reftype);
+use Config qw(%Config);
+use constant is_usethreads => $Config{usethreads};
$Safe::VERSION = "2.19";
@@ -288,8 +291,26 @@
my ($obj, $expr, $strict) = @_;
my $root = $obj->{Root};
- my $evalsub = lexless_anon_sub($root,$strict, $expr);
- return Opcode::_safe_call_sv($root, $obj->{Mask}, $evalsub);
+ my $evalsub = lexless_anon_sub($root, $strict, $expr);
+ my @ret = (wantarray)
+ ? Opcode::_safe_call_sv($root, $obj->{Mask}, $evalsub)
+ : scalar Opcode::_safe_call_sv($root, $obj->{Mask}, $evalsub);
+
+ # RT#60374: Safe.pm sort {} bug with -Dusethreads
+ # If the Safe eval returns a code ref in a perl compiled with usethreads
+ # then wrap code ref with _safe_call_sv so that, when called, the
+ # execution will happen with the compartment fully 'in effect'.
+ # Needed to fix sort blocks that reference $a & $b and
+ # possibly other subtle issues.
+ if (is_usethreads()) {
+ for my $ret (@ret) { # edit (via alias) any CODE refs
+ next unless (reftype($ret)||'') eq 'CODE';
+ my $sub = $ret; # avoid closure problems
+ $ret = sub { Opcode::_safe_call_sv($root, $obj->{Mask}, $sub) };
+ }
+ }
+
+ return (wantarray) ? @ret : $ret[0];
}
sub rdo {
diff -rNu Safe-2.19-orig/t/safesort.t Safe-2.19/t/safesort.t
--- Safe-2.19-orig/t/safesort.t 1970-01-01 01:00:00.000000000 +0100
+++ Safe-2.19/t/safesort.t 2009-11-09 15:53:10.000000000 +0000
@@ -0,0 +1,37 @@
+#!./perl -w
+$|=1;
+BEGIN {
+ if($ENV{PERL_CORE}) {
+ chdir 't' if -d 't';
+ @INC = '../lib';
+ }
+ require Config; import Config;
+ if ($Config{'extensions'} !~ /\bOpcode\b/ && $Config{'osname'} ne 'VMS') {
+ print "1..0\n";
+ exit 0;
+ }
+}
+
+use Safe 1.00;
+use Test::More tests => 4;
+
+my $safe = Safe->new('PLPerl');
+$safe->permit_only(qw(:default sort));
+
+my $func = $safe->reval(<<'EOS');
+
+ # uses quotes in { "$a" <=> $b } to avoid the optimizer replacing the block
+ # with a hardwired comparison
+ { package Pkg; sub p_sort { return sort { "$a" <=> $b } qw(2 1 3); } }
+ sub l_sort { return sort { "$a" <=> $b } qw(2 1 3); }
+
+ return sub { return join(",",l_sort()), join(",",Pkg::p_sort()) }
+
+EOS
+
+is $@, '', 'reval should not fail';
+is ref $func, 'CODE', 'reval should return a CODE ref';
+
+my ($l_sorted, $p_sorted) = $func->();
+is $l_sorted, "1,2,3";
+is $p_sorted, "1,2,3";
|
From david@kineticode.comOn Nov 9, 2009, at 8:21 AM, Tim Bunce wrote:
Perhaps it got lost in the shuffle, and I'm certainly glad to see a Best, David |
From @timbunceOn Mon, Nov 09, 2009 at 09:10:18AM -0800, David E. Wheeler wrote:
It reached the ticket ok:
In http://rt.perl.org/rt3/Ticket/Display.html?id=60374#txn-627795 Tim. |
From @nwc10On Sun, Nov 08, 2009 at 11:36:46PM +0000, Tim Bunce wrote:
The logical way to do this would be to indirect via an entry in the
I think that that doesn't cope with package foo; Nicholas Clark * I think. I don't know this stuff well. |
From @timbunceOn Tue, Nov 10, 2009 at 02:56:36PM +0000, Nicholas Clark wrote:
I couldn't get it to work even without that (but trying to follow the Alex's approach, of arranging to execute the closure 'inside' the On the other hand, the fact it costs two extra sub calls is unfortunate.
Me neither. It's been many years since I helped develop Opcode and Safe. Tim. |
From @timbunceI'm not sure who to ask to apply the patch to Safe.pm. |
From @rgsNow fixed with Safe 2.20. |
@rgs - Status changed from 'open' to 'resolved' |
From @steve-m-haydemerphq wrote on 2009-10-08:
Why do you say that? I always used to use a non-threaded perl on Win32, and only switched to threaded when I had to because mod_perl-2 needs it. Even now I still don't enable PERL_IMPLICIT_SYS because I have no need for the fork() emulation, and prefer Perl's malloc() (which can't be enabled with PERL_IMPLICIT_SYS) to the CRT's version. |
Migrated from rt.perl.org#60374 (status was 'resolved')
Searchable as RT60374$
The text was updated successfully, but these errors were encountered: