Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bleadperl v5.27.3-34-gf6107ca24b breaks MLEHMANN/AnyEvent-HTTP-2.23.tar.gz #16162

Closed
p5pRT opened this issue Sep 21, 2017 · 42 comments
Closed

Bleadperl v5.27.3-34-gf6107ca24b breaks MLEHMANN/AnyEvent-HTTP-2.23.tar.gz #16162

p5pRT opened this issue Sep 21, 2017 · 42 comments
Labels

Comments

@p5pRT
Copy link
Collaborator

@p5pRT p5pRT commented Sep 21, 2017

Migrated from rt.perl.org#132142 (status was 'open')

Searchable as RT132142$

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 21, 2017

From @andk

As usual, cudos to discoverer Slaven!

bisect

commit f6107ca24b4cf22dcf7fd69d65612ad718c48fca
Author: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
Date: Wed Aug 30 22:35:17 2017 +0100

Strengthen weak refs when sorting in-place

diagnostics:
http://www.cpantesters.org/cpan/report/86732923

Perl Info
perl -V
-------
Summary of my perl5 (revision 5 version 27 subversion 4) configuration:
Commit id: ae59d0fba73f3a4b9f082a26fad09e6cb2553dd2
Platform:
osname=linux
osvers=4.9.0-2-amd64
archname=x86_64-linux
uname='linux k93msid 4.9.0-2-amd64 #1 smp debian 4.9.18-1 (2017-03-30) x86_64 gnulinux '
config_args='-Dprefix=/home/sand/src/perl/repoperls/installed-perls/host/k93msid/v5.27.3-41-gae59d0fba7/3c10 -Dmyhostname=k93msid -Dinstallusrbinperl=n -Uversiononly -Dusedevel -des -Ui_db -Dlibswanted=cl pthread socket inet nsl gdbm dbm malloc dl ld sun m crypt sec util c cposix posix ucb BSD gdbm_compat -Uuseithreads -Uuselongdouble -DEBUGGING=none'
hint=recommended
useposix=true
d_sigaction=define
useithreads=undef
usemultiplicity=undef
use64bitint=define
use64bitall=define
uselongdouble=undef
usemymalloc=n
default_inc_excludes_dot=define
bincompat5005=undef
Compiler:
cc='cc'
ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2'
optimize='-O2'
cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
ccversion=''
gccversion='6.3.0 20170406'
gccosandvers=''
intsize=4
longsize=8
ptrsize=8
doublesize=8
byteorder=12345678
doublekind=3
d_longlong=define
longlongsize=8
d_longdbl=define
longdblsize=16
longdblkind=3
ivtype='long'
ivsize=8
nvtype='double'
nvsize=8
Off_t='off_t'
lseeksize=8
alignbytes=8
prototype=define
Linker and Libraries:
ld='cc'
ldflags =' -fstack-protector-strong -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/6/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
libc=libc-2.24.so
so=so
useshrplib=false
libperl=libperl.a
gnulibc_version='2.24'
Dynamic Linking:
dlsrc=dl_dlopen.xs
dlext=so
d_dlsymun=undef
ccdlflags='-Wl,-E'
cccdlflags='-fPIC'
lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'


Characteristics of this binary (from libperl):
Compile-time options:
HAS_TIMES
PERLIO_LAYERS
PERL_COPY_ON_WRITE
PERL_DONT_CREATE_GVSV
PERL_MALLOC_WRAP
PERL_OP_PARENT
PERL_PRESERVE_IVUV
PERL_USE_DEVEL
USE_64_BIT_ALL
USE_64_BIT_INT
USE_LARGE_FILES
USE_LOCALE
USE_LOCALE_COLLATE
USE_LOCALE_CTYPE
USE_LOCALE_NUMERIC
USE_LOCALE_TIME
USE_PERLIO
USE_PERL_ATOF
Built under linux
Compiled at Sep 8 2017 09:50:13
%ENV:
PERL="/tmp/basesmoker-reloperl-OiBV/bin/perl"
PERL5LIB=""
PERL5OPT=""
PERL5_CPANPLUS_IS_RUNNING="23384"
PERL5_CPAN_IS_RUNNING="23384"
PERL_CANARY_STABILITY_NOPROMPT="1"
PERL_MM_USE_DEFAULT="1"
PERL_USE_UNSAFE_INC="1"
@INC:
/home/sand/src/perl/repoperls/installed-perls/host/k93msid/v5.27.3-41-gae59d0fba7/3c10/lib/site_perl/5.27.4/x86_64-linux
/home/sand/src/perl/repoperls/installed-perls/host/k93msid/v5.27.3-41-gae59d0fba7/3c10/lib/site_perl/5.27.4
/home/sand/src/perl/repoperls/installed-perls/host/k93msid/v5.27.3-41-gae59d0fba7/3c10/lib/5.27.4/x86_64-linux
/home/sand/src/perl/repoperls/installed-perls/host/k93msid/v5.27.3-41-gae59d0fba7/3c10/lib/5.27.4
.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 22, 2017

From @ilmari

(Andreas J. Koenig) (via RT) <perlbug-followup@​perl.org> writes​:

bisect
------
commit f6107ca
Author​: Dagfinn Ilmari Mannsåker <ilmari@​ilmari.org>
Date​: Wed Aug 30 22​:35​:17 2017 +0100

Strengthen weak refs when sorting in\-place

diagnostics
-----------
http​://www.cpantesters.org/cpan/report/86732923

AnyEvent is relying on in-place sorting of its array of pending timers
keeping the references weak, and when it doesn't, user code can't cancel
a timer by dropping its reference to it.

Weakening all the references in the array after sorting it fixes the
issue, but that only works for arrays where all the references are weak.
If an array has a mix of strong and weak references one would have to
record which references were weak before the sort, and then go through
the sorted array and re-weaken them afterwards.

With the old behaviour, getting the strengthening effect was trivial​:
one could assign the array to itself before or after the sort, or defeat
the in-place sort altogether with the =()= trick.

This makes me think that despite all the work that's gone into making
the in-place sort optimisation invisible, the weak ref special case
needs to be kept, i.e. the above commit reverted.

I still think sv_rvunweaken() is a useful API function, so the commit
that added that should be kept.

- ilmari
--
"The surreality of the universe tends towards a maximum" -- Skud's Law
"Never formulate a law or axiom that you're not prepared to live with
the consequences of." -- Skud's Meta-Law

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 22, 2017

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 22, 2017

From zefram@fysh.org

Dagfinn Ilmari Mannsaker wrote​:

With the old behaviour, getting the strengthening effect was trivial​:

With the new behaviour, one can fairly trivially get the
weakness-preserving effect by wrapping the possibly-weak references in
an extra layer of reference.

This makes me think that despite all the work that's gone into making
the in-place sort optimisation invisible, the weak ref special case
needs to be kept, i.e. the above commit reverted.

-1. The optimisation should be a pure optimisation. If the useful
behaviour of the old faulty optimisation were somehow not easily
reproduced, it should be made available by explicit means, not kept as
a funny special case of an operator that normally means something else.

-zefram

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 25, 2017

From @andk

Also affected​: MGRIMES/AnyEvent-Filesys-Notify-1.21.tar.gz

Result type​: intermittent failures. Up to v5.27.3-33-gae2cf9f629 I ran
make test in a loop for several minutes and produced only passes, no
fails. With v5.27.3-34-gf6107ca24b and later I could produce both passes
and fails, it was just a matter of running a dozen or so tests to
produce both kinds of result.

--
andreas

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 25, 2017

From @tonycoz

On Fri, Sep 22, 2017 at 03​:48​:18AM +0100, Zefram wrote​:

Dagfinn Ilmari Mannsaker wrote​:

With the old behaviour, getting the strengthening effect was trivial​:

With the new behaviour, one can fairly trivially get the
weakness-preserving effect by wrapping the possibly-weak references in
an extra layer of reference.

This makes me think that despite all the work that's gone into making
the in-place sort optimisation invisible, the weak ref special case
needs to be kept, i.e. the above commit reverted.

-1. The optimisation should be a pure optimisation. If the useful
behaviour of the old faulty optimisation were somehow not easily
reproduced, it should be made available by explicit means, not kept as
a funny special case of an operator that normally means something else.

I agree.

Tony

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 26, 2017

From @ilmari

Zefram <zefram@​fysh.org> writes​:

Dagfinn Ilmari Mannsaker wrote​:

With the old behaviour, getting the strengthening effect was trivial​:

With the new behaviour, one can fairly trivially get the
weakness-preserving effect by wrapping the possibly-weak references in
an extra layer of reference.

Attached is a patch for AnyEvent that does this for its pending timer
list. It's pretty straight-forward, but I'm not in the mood for facing
MLEHMANN's wrath over perl5-porters breaking his code by submitting it
to him.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 26, 2017

From @ilmari

0001-Use-strong-references-to-weak-references-in-the-time.patch
From eb30dd3aab9c33204418c77f7f81d3acb6e5b3eb Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Dagfinn=20Ilmari=20Manns=C3=A5ker?= <ilmari@ilmari.org>
Date: Tue, 26 Sep 2017 22:03:48 +0100
Subject: [PATCH] Use strong references to weak references in the @timer

Perl 5.27.4 fixes the in-place sort optimisation (@a = sort @a) to
strengthen references, like assignment should.

However, we need to keep the references to timers weak, so callers can
cancel them by dropping their reference.  To achieve this, make the
actual references in @timer strong references to weak references to the
value returned to the user, and filter out ones that have gone undef
when necessary.
---
 lib/AnyEvent/Loop.pm | 26 +++++++++++++++-----------
 1 file changed, 15 insertions(+), 11 deletions(-)

diff --git a/lib/AnyEvent/Loop.pm b/lib/AnyEvent/Loop.pm
index fb9351a..f57eab0 100644
--- a/lib/AnyEvent/Loop.pm
+++ b/lib/AnyEvent/Loop.pm
@@ -202,22 +202,26 @@ sub one_event {
    # first sort timers if required (slow)
    if ($MNOW >= $need_sort) {
       $need_sort = 1e300;
-      @timer = sort { $a->[0] <=> $b->[0] } @timer;
+      @timer = sort { $$a->[0] <=> $$b->[0] } grep { defined $$_ } @timer;
+   }
+   else {
+      # shift off any cancelled timers at the start of the queue
+      shift @timer while @timer and not defined ${$timer[0]};
    }
 
    # handle all pending timers
-   if (@timer && $timer[0][0] <= $MNOW) {
+   if (@timer && ${$timer[0]}->[0] <= $MNOW) {
       do {
-         my $timer = shift @timer;
-         $timer->[1] && $timer->[1]($timer);
-      } while @timer && $timer[0][0] <= $MNOW;
+         my $timer = ${shift @timer};
+         $timer && $timer->[1] && $timer->[1]($timer);
+      } while @timer && ${$timer[0]}->[0] <= $MNOW;
 
    } else {
       # poll for I/O events, we do not do this when there
       # were any pending timers to ensure that one_event returns
       # quickly when some timers have been handled
       my ($wait, @vec, $fds)
-         = (@timer && $timer[0][0] < $need_sort ? $timer[0][0] : $need_sort) - $MNOW;
+         = (@timer && ${$timer[0]}->[0] < $need_sort ? ${$timer[0]}->[0] : $need_sort) - $MNOW;
 
       $wait = $wait < MAXWAIT ? $wait + ROUNDUP : MAXWAIT;
       $wait = 0 if @idle;
@@ -252,7 +256,7 @@ sub one_event {
       } elsif (AnyEvent::WIN32 && $fds && $! == AnyEvent::Util::WSAEINVAL) {
          # buggy microshit windoze asks us to route around it
          CORE::select undef, undef, undef, $wait if $wait;
-      } elsif (!@timer || $timer[0][0] > $MNOW && !$fds) {
+      } elsif (!@timer || ${$timer[0]}->[0] > $MNOW && !$fds) {
          $$$_ && $$$_->() for @idle = grep $$$_, @idle;
       }
    }
@@ -319,8 +323,8 @@ sub timer($$$) {
    if ($interval) {
       $self = [$MNOW + $after , sub {
          $_[0][0] = List::Util::max $_[0][0] + $interval, $MNOW;
-         push @timer, $_[0];
-         weaken $timer[-1];
+         weaken(my $weak = $_[0]);
+         push @timer, \$weak;
          $need_sort = $_[0][0] if $_[0][0] < $need_sort;
          &$cb;
       }];
@@ -328,8 +332,8 @@ sub timer($$$) {
       $self = [$MNOW + $after, $cb];
    }
 
-   push @timer, $self;
-   weaken $timer[-1];
+   weaken(my $weakself = $self);
+   push @timer, \$weakself;
    $need_sort = $self->[0] if $self->[0] < $need_sort;
 
    $self
-- 
2.7.4

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 26, 2017

From @ilmari

This makes me think that despite all the work that's gone into making
the in-place sort optimisation invisible, the weak ref special case
needs to be kept, i.e. the above commit reverted.

-1. The optimisation should be a pure optimisation. If the useful
behaviour of the old faulty optimisation were somehow not easily
reproduced, it should be made available by explicit means, not kept as
a funny special case of an operator that normally means something else.

Normally I would agree, but the previous behaviour has been there as
long as in-place sort has, i.e. since 5.10. I also noticed that the
in-place reverse optimisation (since 5.12) also preservers weakness.
Presumably we want to fix that too (as I've done on the
ilmari/unweaken-inplace-reverse branch). Another "inconsistency" is
that in-place reverse preserves the deletedness of array elements.

- ilmari
--
- Twitter seems more influential [than blogs] in the 'gets reported in
  the mainstream press' sense at least. - Matt McLeod
- That'd be because the content of a tweet is easier to condense down
  to a mainstream media article. - Calle Dybedahl

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 27, 2017

From zefram@fysh.org

Dagfinn Ilmari Mannsaker wrote​:

  It's pretty straight\-forward\, but I'm not in the mood for facing

MLEHMANN's wrath over perl5-porters breaking his code by submitting it
to him.

You could just stick it in the RT queue. (And face even more wrath.)

Normally I would agree, but the previous behaviour has been there as
long as in-place sort has, i.e. since 5.10.

It's been a bug all that time.

in-place reverse optimisation (since 5.12) also preservers weakness.
Presumably we want to fix that too

Yes.

                                     Another "inconsistency" is

that in-place reverse preserves the deletedness of array elements.

That's not detectable, is it? Preserving deletedness seems like it
would just avoid allocating some unnecessary SVs.

-zefram

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Sep 27, 2017

From @ilmari

Zefram <zefram@​fysh.org> writes​:

Dagfinn Ilmari Mannsaker wrote​:

                                     Another "inconsistency" is

that in-place reverse preserves the deletedness of array elements.

That's not detectable, is it? Preserving deletedness seems like it
would just avoid allocating some unnecessary SVs.

$ perl -E '@​a=(1..3); delete $a[1]; @​a = reverse @​a; say 0+exists $a[1]'
0

--
- Twitter seems more influential [than blogs] in the 'gets reported in
  the mainstream press' sense at least. - Matt McLeod
- That'd be because the content of a tweet is easier to condense down
  to a mainstream media article. - Calle Dybedahl

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 17, 2017

From @andk

What is the state of this ticket? From looking at it it looks stalled.
Was there any additional activity?

Has anybody summarised? Reported to Marc? Provided a patch? Considered
improving? Considered reverting?

Thanks!
--
andreas

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 18, 2017

From @ilmari

Andreas Koenig <andreas.koenig.7os6VVqR@​franz.ak.mind.de> writes​:

What is the state of this ticket? From looking at it it looks stalled.
Was there any additional activity?

Has anybody summarised? Reported to Marc? Provided a patch? Considered
improving? Considered reverting?

I provided a patch in
https://rt.perl.org/Public/Bug/Display.html?id=132142#txn-1493873, but
just realised there's a simpler way to do it, which I've attached.

As mentioned in that previous message, I'm not in the mood to confront
Marc over this, but Sawyer graciously offered to do that for me.

Sawyer, do you feel like sending these two patches to Marc and suggest
he applies one of them to keep AnyEvent working on blead? Note that
AnyEvent's own test suite doesn't trigger the bug, but AnyEvent​::HTTP
does.

- ilmari
--
- Twitter seems more influential [than blogs] in the 'gets reported in
  the mainstream press' sense at least. - Matt McLeod
- That'd be because the content of a tweet is easier to condense down
  to a mainstream media article. - Calle Dybedahl

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 18, 2017

From @ilmari

0001-Re-weaken-timer-references-after-sorting.patch
From 8e12a666543e50043419630d5e4ff00c7dd277e2 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Dagfinn=20Ilmari=20Manns=C3=A5ker?= <ilmari@ilmari.org>
Date: Fri, 17 Nov 2017 23:59:30 +0000
Subject: [PATCH] Re-weaken timer references after sorting

Perl 5.27.4 fixes the in-place sort optimisation (@a = sort @a) to
strengthen references, like assignment should.

However, we need to keep the references to timers weak, so callers can
cancel them by dropping their reference.
---
 lib/AnyEvent/Loop.pm | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/lib/AnyEvent/Loop.pm b/lib/AnyEvent/Loop.pm
index fb9351a..f7145fd 100644
--- a/lib/AnyEvent/Loop.pm
+++ b/lib/AnyEvent/Loop.pm
@@ -173,6 +173,9 @@ BEGIN {
    $round = 0.001 if $round < 0.001; # 1ms is enough for us
    $round -= $round * 1e-2; # 0.1 => 0.099
    eval "sub ROUNDUP() { $round }";
+
+   my $unweakens = $] >= 5.027004 ? 1 : 0;
+   eval "sub INPLACE_SORT_UNWEAKENS() { $unweakens }";
 }
 
 _update_clock;
@@ -203,6 +206,9 @@ sub one_event {
    if ($MNOW >= $need_sort) {
       $need_sort = 1e300;
       @timer = sort { $a->[0] <=> $b->[0] } @timer;
+      if (INPLACE_SORT_UNWEAKENS) {
+          weaken($_) for @timer;
+      }
    }
 
    # handle all pending timers
-- 
2.7.4

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Nov 19, 2017

From @xsawyerx

On Fri, 17 Nov 2017 16​:09​:12 -0800, ilmari wrote​:

Andreas Koenig <andreas.koenig.7os6VVqR@​franz.ak.mind.de> writes​:

What is the state of this ticket? From looking at it it looks stalled.
Was there any additional activity?

Has anybody summarised? Reported to Marc? Provided a patch? Considered
improving? Considered reverting?

I provided a patch in
https://rt.perl.org/Public/Bug/Display.html?id=132142#txn-1493873, but
just realised there's a simpler way to do it, which I've attached.

As mentioned in that previous message, I'm not in the mood to confront
Marc over this, but Sawyer graciously offered to do that for me.

Sawyer, do you feel like sending these two patches to Marc and suggest
he applies one of them to keep AnyEvent working on blead? Note that
AnyEvent's own test suite doesn't trigger the bug, but AnyEvent​::HTTP
does.

I didn't receive an email for this correspondence so I only saw the ping from Andreas. I have just sent an email to Marc with the latest patch, asking to apply the patch and release a new version. P5P was CC'ed on the email as well.

Thank you for producing the patches, Ilmari.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Apr 19, 2018

From @iabyn

On Sun, Nov 19, 2017 at 06​:10​:28AM -0800, Sawyer X via RT wrote​:

On Fri, 17 Nov 2017 16​:09​:12 -0800, ilmari wrote​:

Andreas Koenig <andreas.koenig.7os6VVqR@​franz.ak.mind.de> writes​:

What is the state of this ticket? From looking at it it looks stalled.
Was there any additional activity?

Has anybody summarised? Reported to Marc? Provided a patch? Considered
improving? Considered reverting?

I provided a patch in
https://rt.perl.org/Public/Bug/Display.html?id=132142#txn-1493873, but
just realised there's a simpler way to do it, which I've attached.

As mentioned in that previous message, I'm not in the mood to confront
Marc over this, but Sawyer graciously offered to do that for me.

Sawyer, do you feel like sending these two patches to Marc and suggest
he applies one of them to keep AnyEvent working on blead? Note that
AnyEvent's own test suite doesn't trigger the bug, but AnyEvent​::HTTP
does.

I didn't receive an email for this correspondence so I only saw the ping from Andreas. I have just sent an email to Marc with the latest patch, asking to apply the patch and release a new version. P5P was CC'ed on the email as well.

My understanding of the current state of this 5.28 blocker ticket is​:

1) A bug fix to blead removed an inadvertent user-visible difference
between '@​b = sort @​a' and '@​a sort @​a'; the latter is internally optimised,
but such optimisations shouldn't make visible changes to sort behaviour.

2) AnyEvent relied on this now-fixed buggy behaviour;

3) A patch has been privately submitted to Marc;

4) There has not been a new release of AnyEvent yet;

5) There was a brief discussion about the desirability of an explicit
in-place sort feature.

I propose

a) this ticket is removed from 5.28 blockers.
b) is kept open in case anyone wants to propose an in-place sort syntax or
mechanism.

--
Hofstadter's Law​: It always takes longer than you expect, even when you
take into account Hofstadter's Law.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Apr 20, 2018

From @andk

On Thu, 19 Apr 2018 14​:48​:47 -0700, "Dave Mitchell via RT" <perlbug-followup@​perl.org> said​:

  > On Sun, Nov 19, 2017 at 06​:10​:28AM -0800, Sawyer X via RT wrote​:

On Fri, 17 Nov 2017 16​:09​:12 -0800, ilmari wrote​:

Andreas Koenig <andreas.koenig.7os6VVqR@​franz.ak.mind.de> writes​:

What is the state of this ticket? From looking at it it looks stalled.
Was there any additional activity?

Has anybody summarised? Reported to Marc? Provided a patch? Considered
improving? Considered reverting?

I provided a patch in
https://rt.perl.org/Public/Bug/Display.html?id=132142#txn-1493873, but
just realised there's a simpler way to do it, which I've attached.

As mentioned in that previous message, I'm not in the mood to confront
Marc over this, but Sawyer graciously offered to do that for me.

Sawyer, do you feel like sending these two patches to Marc and suggest
he applies one of them to keep AnyEvent working on blead? Note that
AnyEvent's own test suite doesn't trigger the bug, but AnyEvent​::HTTP
does.

I didn't receive an email for this correspondence so I only saw the
ping from Andreas. I have just sent an email to Marc with the latest
patch, asking to apply the patch and release a new version. P5P was
CC'ed on the email as well.

  > My understanding of the current state of this 5.28 blocker ticket is​:

  > 1) A bug fix to blead removed an inadvertent user-visible difference
  > between '@​b = sort @​a' and '@​a sort @​a'; the latter is internally optimised,
  > but such optimisations shouldn't make visible changes to sort
  > behaviour.

It has been pointed out that the "internally optimized behaviour" of '@​a
= sort @​a' was documented as a way to get an in-place sort. It was
established behaviour for a long time. Given that it was both proudly
announced and well established, I find the term "bug fix" for its
removal problematic. This is a change in bahaviour. As such it may be a
bug fix for one party and a breakage for the other. Fair discussion
should try to balance the thinking and take this into account.

  > 2) AnyEvent relied on this now-fixed buggy behaviour;

The problematic terms continue.

  > 3) A patch has been privately submitted to Marc;

But the patch has issues. Discussion about it has been avoided and
prevented.

  > 4) There has not been a new release of AnyEvent yet;

What could that probably mean?

  > 5) There was a brief discussion about the desirability of an explicit
  > in-place sort feature.

Why was it so brief? Originally the in-place sort was deemed as a
feature. To remove it, is now called a bug fix. I would say seeking for
solutions with the decisive extra bit of effort should be possible.

  > I propose

  > a) this ticket is removed from 5.28 blockers.
  > b) is kept open in case anyone wants to propose an in-place sort syntax or
  > mechanism.

I would suggest a revert with a chance to retry for 5.30, but with the
mandate to try to find a way to retain in-place sort.

  > Hofstadter's Law​: It always takes longer than you expect, even when you
  > take into account Hofstadter's Law.

Long live Hofstadter,
--
andreas

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Apr 20, 2018

From tom@binary.com

On 20 April 2018 at 12​:50, Andreas Koenig <
andreas.koenig.7os6VVqR@​franz.ak.mind.de> wrote​:

It has been pointed out that the "internally optimized behaviour" of '@​a
= sort @​a' was documented as a way to get an in-place sort. It was
established behaviour for a long time. Given that it was both proudly
announced and well established, I find the term "bug fix" for its
removal problematic. This is a change in bahaviour. As such it may be a
bug fix for one party and a breakage for the other. Fair discussion
should try to balance the thinking and take this into account.

I believe the original release note from perl5100delta was​:

In-place sorting
Sorting arrays in place ("@​a = sort @​a") is now optimized to avoid
making
a temporary copy of the array.

Likewise, "reverse sort ..." is now optimized to sort in reverse,
avoiding
the generation of a temporary intermediate list.

Since there was no corresponding documentation update to sort itself, and
no mention of how this would affect magic or weakrefs, I think it would be
reasonable to conclude that this is intended purely as a non-user-visible
optimisation? If the behaviour is intentional, then why (in 5.26) does it
only happen for sort, but not the reverse sort that was explicitly called
out in the same release note?

2) AnyEvent relied on this now-fixed buggy behaviour;

The problematic terms continue.

The sort optimisation was added in 5.10. AnyEvent has test passes (and code
mention of) 5.8.0. If that's correct - I haven't looked at the relevant
code to confirm - then I think relying on in-place sort optimisation could
rightly be described as a bug?

When I mentioned the in-place sort issue to ilmari originally, I was under
the impression that it was a well-known bug that no one had ever got around
to fixing (similar to the case with plain reverse).

It's something I've run into a few times in the last few years, is more of
an annoyance than a genuine complaint, but does seem like an arbitrary
behavioural difference that should be discouraged rather than documented.
To me, it's very clearly a bug, and inconsistent​: we have a fine example of
in-place modification in chomp(), I'd expect similar syntax for an in-place
sort or reverse feature.

cheers,

Tom

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Apr 20, 2018

From @iabyn

On Fri, Apr 20, 2018 at 06​:50​:54AM +0200, Andreas Koenig wrote​:

It has been pointed out that the "internally optimized behaviour" of '@​a
= sort @​a' was documented as a way to get an in-place sort.

Again, as the author of that optimisation,
1) it was my intention that it should make no user-visible difference
  apart from speed;
2) if I had been aware of the weakrefs behaviour at the time, I would
  have implemented the optimisation in a way that strengthened weak refs;
3) there is no documentation for the sort function which describes this
behaviour. The 5.10.1 perldelta mentions in-place sorting in the
'Performance Enhancements' section, and makes no particular claim about
whether weak refs are strengthened, or any implication that a change in
behaviour is intended.

It was
established behaviour for a long time.

Any bug fix must by definition change established behaviour.

Given that it was both proudly
announced and well established,

We'll have to disagree about the "announced" bit.

I find the term "bug fix" for its
removal problematic. This is a change in bahaviour. As such it may be a
bug fix for one party and a breakage for the other. Fair discussion
should try to balance the thinking and take this into account.

My personal feeling is that avoiding unexpected differences in behaviour
between '@​a = sort @​a' and '@​b = sort @​a' outweighs the cost of the change
in behaviour. Other people may feel differently.

3) A patch has been privately submitted to Marc;

But the patch has issues. Discussion about it has been avoided and
prevented.

Unusually for CPAN authors, Marc doesn't use any publicly-facing bug
tracking system for his modules, so it is hard to have a discussions about
such issues. Even when Marc isn't banned from this list, it is extremely
difficult to have any sort of technical discussion with him, since he
often resorts to personal attacks. I have, on this list, been repeatedly
called a liar by him. When I submitted a patch to him for one of his
modules, his initial reply was to tell me that my attitude was atrocious.
This sort of behaviour is a strong disincentive for anyone to engage with
him.

4) There has not been a new release of AnyEvent yet;

What could that probably mean?

Again, due to the lack of a public bug tracker, that's hard to guess.
Perhaps it's a poor patch; perhaps it a good patch which Marc hasn't had
time to apply yet?

5) There was a brief discussion about the desirability of an explicit
in-place sort feature.

Why was it so brief?

Presumably because no-one could come up with any good suggestions.

Originally the in-place sort was deemed as a
feature.

Again, I disagree.

To remove it, is now called a bug fix. I would say seeking for
solutions with the decisive extra bit of effort should be possible.

Many things are possible. In practice, perl gets 1000 new RT tickets per
year, plus a further 150 security tickets; there are many things which
occupy our time and energy.

I would suggest a revert with a chance to retry for 5.30, but with the
mandate to try to find a way to retain in-place sort.

Does anyone else here any feelings either way?

--
"I do not resent criticism, even when, for the sake of emphasis,
it parts for the time with reality".
  -- Winston Churchill, House of Commons, 22nd Jan 1941.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Apr 25, 2018

From @iabyn

On Fri, Apr 20, 2018 at 08​:29​:08AM +0100, Dave Mitchell wrote​:

On Fri, Apr 20, 2018 at 06​:50​:54AM +0200, Andreas Koenig wrote​:
[ snip discussion about @​a = sort @​a optimisation not converting weak
ref back to strong refs ]

I would suggest a revert with a chance to retry for 5.30, but with the
mandate to try to find a way to retain in-place sort.

Does anyone else here any feelings either way?

From the silence, I'm assuming no-one else wants this change reverted
before 5.28.

--
A power surge on the Bridge is rapidly and correctly diagnosed as a faulty
capacitor by the highly-trained and competent engineering staff.
  -- Things That Never Happen in "Star Trek" #9

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Apr 25, 2018

From @khwilliamson

On 04/25/2018 04​:18 AM, Dave Mitchell wrote​:

On Fri, Apr 20, 2018 at 08​:29​:08AM +0100, Dave Mitchell wrote​:

On Fri, Apr 20, 2018 at 06​:50​:54AM +0200, Andreas Koenig wrote​:
[ snip discussion about @​a = sort @​a optimisation not converting weak
ref back to strong refs ]

I would suggest a revert with a chance to retry for 5.30, but with the
mandate to try to find a way to retain in-place sort.

Does anyone else here any feelings either way?

From the silence, I'm assuming no-one else wants this change reverted
before 5.28.

I could support a temporary revert.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Apr 25, 2018

From @jkeenan

On Wed, 25 Apr 2018 10​:20​:08 GMT, davem wrote​:

On Fri, Apr 20, 2018 at 08​:29​:08AM +0100, Dave Mitchell wrote​:

On Fri, Apr 20, 2018 at 06​:50​:54AM +0200, Andreas Koenig wrote​:
[ snip discussion about @​a = sort @​a optimisation not converting weak
ref back to strong refs ]

I would suggest a revert with a chance to retry for 5.30, but with the
mandate to try to find a way to retain in-place sort.

Does anyone else here any feelings either way?

From the silence, I'm assuming no-one else wants this change reverted
before 5.28.

We're so close to release that I think reverting would open up more problems than it would solve. MLEHMANN and his defenders are going to complain regardless of what we do. We've lived with the code in its current state, so we have a fair idea of what its impact will be. Unless someone wants to revert in a branch and test most of CPAN against that branch, and unless such a process has demonstrably better results than the current state of blead, I think we should proceed on course.

Let's face the music and dance.

Jim Keenan

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Apr 26, 2018

From @andk

On Wed, 25 Apr 2018 12​:59​:11 -0700, "James E Keenan via RT" <perlbug-followup@​perl.org> said​:

  > MLEHMANN and his defenders are going to complain regardless of what
  > we do.

While you try to make this an argument about my gang vs your gang,
something much more important is at stake. We're not arguing about the
author of that test, we're arguing about impacts of breaking code
written in the language that made CPAN possible on the userbase thereof.
Please keep that in mind.

Thank you very much,
--
andreas

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Apr 26, 2018

From @iabyn

On Wed, Apr 25, 2018 at 01​:45​:46PM -0600, Karl Williamson wrote​:

On 04/25/2018 04​:18 AM, Dave Mitchell wrote​:

On Fri, Apr 20, 2018 at 08​:29​:08AM +0100, Dave Mitchell wrote​:

On Fri, Apr 20, 2018 at 06​:50​:54AM +0200, Andreas Koenig wrote​:
[ snip discussion about @​a = sort @​a optimisation not converting weak
ref back to strong refs ]

I would suggest a revert with a chance to retry for 5.30, but with the
mandate to try to find a way to retain in-place sort.

Does anyone else here any feelings either way?

From the silence, I'm assuming no-one else wants this change reverted
before 5.28.

I could support a temporary revert.

I have pushed a branch, smoke-me/davem/weak_sort, which contains (I hope)
a revert of the relevant commits, in case we decide to go ahead.

--
"Procrastination grows to fill the available time"
  -- Mitchell's corollary to Parkinson's Law

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Apr 26, 2018

From @xsawyerx

Alright, everyone. It's time. I'm calling it.

We revert this fix for 5.28 and leave it for 5.30.

I have a lot to say about this and I'm incredibly frustrated and unhappy
with this situation and the conclusion we have to reach, but I think at
this point it's the right call.

I have privately emailed Dave about this and he now provided the branch.
Thank you, Dave.

On 04/26/2018 07​:21 AM, Andreas Koenig wrote​:

On Wed, 25 Apr 2018 12​:59​:11 -0700, "James E Keenan via RT" <perlbug-followup@​perl.org> said​:
MLEHMANN and his defenders are going to complain regardless of what
we do.

While you try to make this an argument about my gang vs your gang,
something much more important is at stake. We're not arguing about the
author of that test, we're arguing about impacts of breaking code
written in the language that made CPAN possible on the userbase thereof.
Please keep that in mind.

Thank you very much,

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Apr 27, 2018

From @iabyn

On Thu, Apr 26, 2018 at 02​:51​:06PM +0300, Sawyer X wrote​:

We revert this fix for 5.28 and leave it for 5.30.

Revert now merged into blead via d830645

--
I thought I was wrong once, but I was mistaken.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jun 10, 2018

From @andk

After some digging I finally found the original tickets that started
this story. Enjoy the timeline below that now provides the for so long
missing piece. It would be cool if it could help this issue receive the
appropriate amount of fairness.

2004-01-25​:

https://rt.perl.org/Public/Bug/Display.html?id=25263

RT issue
From​: Dan Jacobson
Subject​: sort -u
Summary​: A feature request for 'sort -u'

2004-01-26​:

https://rt.perl.org/Public/Bug/Display.html?id=25263#txn-72227

Within same RT ticket
From​: Tels
Summary​: adding feature request for sorting in-place

2004-02-20​:

https://rt.perl.org/Public/Bug/Display.html?id=25263#txn-79013

Within same RT ticket
From​: Dave Mitchell
Citation​: "I've now applied change #22349 to bleed, which makes @​a =
  sort @​a act inplace."

2017-09-21​:

https://rt.perl.org/Public/Bug/Display.html?id=132142

BBC ticket
From​: Andreas Koenig
Subject​: Bleadperl v5.27.3-34-gf6107ca24b breaks MLEHMANN/AnyEvent-HTTP-2.23.tar.gz

2017-11-19​:

https://www.nntp.perl.org/group/perl.perl5.porters/2017/12/msg248259.html

From​: Sawyer X
Subject​: Re​: Patch for AnyEvent​::HTTP
Citations​: "This bug will stay fixed."

2018-04-26​:

https://rt.perl.org/Public/Bug/Display.html?id=132142#txn-1550198

From​: Sawyer X
Citation​: "We revert this fix for 5.28 and leave it for 5.30."

--
andreas

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jun 10, 2018

From @iabyn

On Sun, Jun 10, 2018 at 08​:52​:25PM +0200, Andreas Koenig wrote​:

After some digging I finally found the original tickets that started
this story. Enjoy the timeline below that now provides the for so long
missing piece. It would be cool if it could help this issue receive the
appropriate amount of fairness.

I am genuinely puzzled by your email. In what way do you feel that this
issue is currently not receiving "the appropriate amount of fairness",
and in what way do you feel that the provided links help in improving
fairness?

In particular, reading through the 2004 ticket in its entirety only
confirms in my mind that the work I did on in-place sort back then was
intended as an internal optimisation to save memory memory and CPU, and I
am still of the opinion that if, back in 2004 someone had pointed out that
my optimisation failed to strengthen weak refs, then I would have regarded
it as a bug and fixed it.

--
Technology is dominated by two types of people​: those who understand what
they do not manage, and those who manage what they do not understand.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jun 24, 2018

From @andk

On Sun, 10 Jun 2018 14​:00​:03 -0700, "Dave Mitchell via RT" <perlbug-followup@​perl.org> said​:

  > On Sun, Jun 10, 2018 at 08​:52​:25PM +0200, Andreas Koenig wrote​:

After some digging I finally found the original tickets that started
this story. Enjoy the timeline below that now provides the for so long
missing piece. It would be cool if it could help this issue receive the
appropriate amount of fairness.

  > I am genuinely puzzled by your email. In what way do you feel that this
  > issue is currently not receiving "the appropriate amount of fairness",
  > and in what way do you feel that the provided links help in improving
  > fairness?

Thanks for asking. I would suggest re-reading the one sentence​:

  : It would be cool if I could say​: sort @​array; or @​array = sort_in_place
  : @​array; and get an inplace sort

Ask yourself what in-place-sort means and answer the following
questions​:

- does an in-place-sort change any element within the array?
- is an in-place-sort conceptually an assignment?
- did the as per 2004-02-20 committed solution provide an in-place-sort
  disguised as an assignment?
- was it clever to provide an in-place-sort disguised as an assignement?

My tentative answers​:

- no
- no
- yes
- probably not

  > In particular, reading through the 2004 ticket in its entirety only
  > confirms in my mind that the work I did on in-place sort back then was
  > intended as an internal optimisation to save memory memory and CPU, and I
  > am still of the opinion that if, back in 2004 someone had pointed out that
  > my optimisation failed to strengthen weak refs, then I would have regarded
  > it as a bug and fixed it.

There is no doubt that your intentions back then and today are
consistent. But a more complete view on the whole matter should have
revealed by now that there are at least two ways to read and argue the
story.

I sincerely believe it cannot be too hard for an average bystander of
good will to argue both ways.

--
andreas

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jun 25, 2018

From @iabyn

On Sun, Jun 24, 2018 at 11​:34​:30AM +0200, Andreas Koenig wrote​:

There is no doubt that your intentions back then and today are
consistent. But a more complete view on the whole matter should have
revealed by now that there are at least two ways to read and argue the
story.

I sincerely believe it cannot be too hard for an average bystander of
good will to argue both ways.

The basic question question is whether a perl programmer could have a
reasonable belief that sort has a special-cased and supported feature, in
that the specific syntax

  @​array1 = sort @​array1

behaves differently than, e.g.

  @​array2 = sort @​array1; @​array1 = @​array1

as regards things like strengthening weak references.

Since there is absolutely no mention of this in the documentation for
sort, there would be an extremely strong assumption of no.

There is an entry in perldelta talking about an 'in-place' optimisation,
with no mention of weak refs etc.

Finally there is the RT ticket, originally concerning a 'sort -u' feature
request, where at one point, Tels suggested, in the context of *saving
memory*, that it would be nice to have a in-place sort feature, for which
he proposed a distinct new syntax (which was not taken up).

I really cannot see how any of this justifies not strengthening weak refs
as being anything but a bug.

--
"Procrastination grows to fill the available time"
  -- Mitchell's corollary to Parkinson's Law

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jun 27, 2018

From @andk

On Mon, 25 Jun 2018 01​:15​:33 -0700, "Dave Mitchell via RT" <perlbug-followup@​perl.org> said​:

  > On Sun, Jun 24, 2018 at 11​:34​:30AM +0200, Andreas Koenig wrote​:

In your previous posting you asked me "in what way do you feel that the
provided links help in improving fairness". In my response I asked you 4
questions. I would be interesting what your answers to the four
questions are. And why did they not help you to argue it both ways?

Thank you very much.
--
andreas

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jun 27, 2018

From @iabyn

On Wed, Jun 27, 2018 at 08​:21​:43AM +0200, Andreas Koenig wrote​:

On Mon, 25 Jun 2018 01​:15​:33 -0700, "Dave Mitchell via RT" <perlbug-followup@​perl.org> said​:

On Sun, Jun 24, 2018 at 11​:34​:30AM +0200, Andreas Koenig wrote​:

In your previous posting you asked me "in what way do you feel that the
provided links help in improving fairness". In my response I asked you 4
questions. I would be interesting what your answers to the four
questions are. And why did they not help you to argue it both ways?

Ask yourself what in-place-sort means and answer the following
questions​:

- does an in-place-sort change any element within the array?
- is an in-place-sort conceptually an assignment?
- did the as per 2004-02-20 committed solution provide an in-place-sort
disguised as an assignment?

- was it clever to provide an in-place-sort disguised as an assignement?

My tentative answers​:

- no
- no
- yes
- probably not

There is ambiguity in the term 'in-place' - it can either refer to
something at the user level, or to describe an internal optimisation.

If the phrase 'in-place' was being used to describe a user-level feature,
then my answers would be​:

  no
  no
  no
  no

If the phrase 'in-place' was being used to describe the implementation
details of an optimisation, then

  yes
  yes
  no
  N/A

NB I am of the firm opinion that the 2004-02-20 commit was entirely
related to the second interpretation.

Here's a question for you. What in the documentation for sort, might lead
a reasonable programmer to assume that @​a = sort @​a is intended behave
specially and (amongst other things), not strengthen weak references?

--
Diplomacy is telling someone to go to hell in such a way that they'll
look forward to the trip

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jul 5, 2018

From @andk

On Wed, 27 Jun 2018 01​:09​:40 -0700, "Dave Mitchell via RT" <perlbug-followup@​perl.org> said​:

  > On Wed, Jun 27, 2018 at 08​:21​:43AM +0200, Andreas Koenig wrote​:

On Mon, 25 Jun 2018 01​:15​:33 -0700, "Dave Mitchell via RT" <perlbug-followup@​perl.org> said​:

On Sun, Jun 24, 2018 at 11​:34​:30AM +0200, Andreas Koenig wrote​:

In your previous posting you asked me "in what way do you feel that the
provided links help in improving fairness". In my response I asked you 4
questions. I would be interesting what your answers to the four
questions are. And why did they not help you to argue it both ways?

Ask yourself what in-place-sort means and answer the following
questions​:

- does an in-place-sort change any element within the array?
- is an in-place-sort conceptually an assignment?
- did the as per 2004-02-20 committed solution provide an in-place-sort
disguised as an assignment?

- was it clever to provide an in-place-sort disguised as an assignement?

My tentative answers​:

- no
- no
- yes
- probably not

  > There is ambiguity in the term 'in-place' - it can either refer to
  > something at the user level, or to describe an internal
  > optimisation.

An ambiguity there is, what a great step forward, thank you!

  > If the phrase 'in-place' was being used to describe a user-level feature,
  > then my answers would be​:

  > no
  > no
  > no

Could you extend on that? Especially, how big is the difference between
an in-place-sort and the solution provided by the February 2004 commit?

I mean, look how perfectly well allocated the elements of the array are
before and after the sort​:

% perl -le 'use Devel​::Peek; @​a = qw(2 1);Dump @​a;@​a = sort @​a;Dump @​a;' 2>&1 | grep at
SV = PVAV(0x561c2532bf08) at 0x561c253493d8
  SV = PV(0x561c2532ada0) at 0x561c2532a1e0
  SV = PV(0x561c2532ae80) at 0x561c2532a3f0
SV = PVAV(0x561c2532bf08) at 0x561c253493d8
  SV = PV(0x561c2532ae80) at 0x561c2532a3f0
  SV = PV(0x561c2532ada0) at 0x561c2532a1e0

Both head and body of every PV involved stay the same. This you can't
get with an ordinary assignment.

  > no

  > If the phrase 'in-place' was being used to describe the implementation
  > details of an optimisation, then

  > yes
  > yes
  > no
  > N/A

  > NB I am of the firm opinion that the 2004-02-20 commit was entirely
  > related to the second interpretation.

But let's recall, that there is a finally acknowledged ambiguity for the
reader of the cited thread
(https://rt.perl.org/Public/Bug/Display.html?id=25263#txn-72227);

  > Here's a question for you. What in the documentation for sort, might lead
  > a reasonable programmer to assume that @​a = sort @​a is intended behave
  > specially and (amongst other things), not strengthen weak references?

It has already been answered above that it was at least documented in
perl5100delta with this text​:

: =head2 In-place sorting
:
: Sorting arrays in place (C<@​a = sort @​a>) is now optimized to avoid
: making a temporary copy of the array.

But there is more to it. Documentation is always more than what projects
provide as the manpages. What is being discussed on mailing lists and in
RT tickets is also relevant for the interpretation of the development of
a language. Much more so in 2004 than in 2018 one might add. And finally
Perl is a leading part of the open source culture, which is one of the
reasons we use and love it. So it *may* have happenen that people picked
up partially from discussions and partially from reading the sources.
I'm not saying this is what happened, I don't know. I believe the
discussion in the ticket and the announcement on perl5100delta and some
small experiments with Devel​::Peek as I showed above can give a
reasonable picture of the state of the feature.

To sum up​: a reasonable perl programmer who was following the
developement on perl5-porters around the time of perl 5.8.4 might have
understood that this in-place sorting was a powerful feature. Readers
who were not reading perl5-porters at that time, can now read the above
linked thread and try to understand how the reasonable programmer might
have taken it up. The reasonable programmer might not have noticed that
the feature did not find its way into the documentation. He might have
used it in a module and uploaded it to CPAN. In fact this did happen
indeed.

--
andreas

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jul 5, 2018

From @iabyn

On Thu, Jul 05, 2018 at 09​:57​:22PM +0200, Andreas Koenig wrote​:

To sum up​: a reasonable perl programmer who was following the
developement on perl5-porters around the time of perl 5.8.4 might have
understood that this in-place sorting was a powerful feature. Readers
who were not reading perl5-porters at that time, can now read the above
linked thread and try to understand how the reasonable programmer might
have taken it up. The reasonable programmer might not have noticed that
the feature did not find its way into the documentation. He might have
used it in a module and uploaded it to CPAN. In fact this did happen
indeed.

This is just beyond absurd. If we apply that criteria to perl generally,
there is essentially no bug we can fix, no change we can ever make, in
case someone made a comment on a p5p thread 14 years ago, which by
squinting hard enough could be interpreted as supporting the buggy
behaviour.

So hooray, perl is complete. We can all go away and work on other things
now.

A *reasonable* perl programmer would have looked at the documentation for
sort, noted that there was absolutely no mention of "in-place" sorting or
@​a = sort @​a having special behaviour, and either concluded that no
special behaviour was intended, or enquired with p5p as to why the
documentation was lacking, and whether this was an oversight.

PS - I don't think for one minute the use of the phrase "in-place" was
ambiguous *in context*, in either the p5p discussion or the perldelta; it
was only ambiguous in the sense that your specific questions about it
provided no context as to what behaviour was being referring to.

--
A walk of a thousand miles begins with a single step...
then continues for another 1,999,999 or so.

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jul 9, 2018

From @ilmari

Dave Mitchell <davem@​iabyn.com> writes​:

A *reasonable* perl programmer would have looked at the documentation for
sort, noted that there was absolutely no mention of "in-place" sorting or
@​a = sort @​a having special behaviour, and either concluded that no
special behaviour was intended, or enquired with p5p as to why the
documentation was lacking, and whether this was an oversight.

FWIW​: My original change came abuot exactly because someone on IRC was
surprised by the undocumented special behaviour of in-place sort not
strengthening weakrefs like a normal array assignment or non-in-place
sort would.

- ilmari
--
- Twitter seems more influential [than blogs] in the 'gets reported in
  the mainstream press' sense at least. - Matt McLeod
- That'd be because the content of a tweet is easier to condense down
  to a mainstream media article. - Calle Dybedahl

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jul 14, 2018

From @andk

On Thu, 5 Jul 2018 22​:38​:24 +0100, Dave Mitchell <davem@​iabyn.com> said​:

  > A *reasonable* perl programmer would have looked at the documentation for
  > sort, noted that there was absolutely no mention of "in-place" sorting or
  > @​a = sort @​a having special behaviour, and either concluded that no
  > special behaviour was intended, or enquired with p5p as to why the
  > documentation was lacking, and whether this was an oversight.

I'm wary whenever somebody argues that something would have needed to be
done by somebody else.

Especially so, when this somebody is a user of the language that the
speaker maintains. Being a perl5-porter means being subject to a higher
standard to what reasonable behaviour is than for a user of the
language. In other words, a porter can never blame a user for
unsatisficing documentation.

We have a case that what the language maintainer demands from the
language user has not happened. But it is never too late to adjust
documentation.

I would argue that

- by having the documentation in the delta manpage,
- not taking it over into the perfunc/sort manpage and
- not noticing the missing part of the documentation for 13 years

there is a case of a customary right on the side of all users of the
language to keep the behaviour as implemented. Especially so, as the
behaviour is useful and actively deployed in a high profile CPAN module
that has a high number of direct and indirect users.

Alternatives have not been suggested yet and might exist that might lead
to an even better solution for all parties.

  > PS - I don't think for one minute the use of the phrase "in-place" was
  > ambiguous *in context*, in either the p5p discussion or the perldelta; it
  > was only ambiguous in the sense that your specific questions about it
  > provided no context as to what behaviour was being referring to.

I'm not sure I can parse this sentence correctly. I think I have
provided quite some context. If you think otherwise, feel free to ask.

--
andreas

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jul 14, 2018

From @iabyn

On Sat, Jul 14, 2018 at 10​:15​:11AM +0200, Andreas Koenig wrote​:

I would argue that

- by having the documentation in the delta manpage,

There was no such documentation in perldelta. There was mention of a new
optimisation, not of a new sort language feature.

There is a strong ethos within core development that new optimisations
shouldn't change user-visible behaviour where possible. If they do,
that's usually considered a bug, or if the trade off is worth it,
something to be documented as a gotcha.

Note also that the change was back-ported to 5.8.4 - an odd thing to do
for a changed language feature.

- not taking it over into the perfunc/sort manpage and

There was nothing to take over.

- not noticing the missing part of the documentation for 13 years

There was nothing missing.

there is a case of a customary right on the side of all users of the
language to keep the behaviour as implemented.

So do you agree that the behaviour which has been present since 5.000 and
was accidentally changed in 5.8.4, should be reverted to the 5.000 state,
since users have an expectation that the implemented behaviour should be
kept?

Alternatives have not been suggested yet and might exist that might lead
to an even better solution for all parties.

I'm open to suggestions.

PS - I don't think for one minute the use of the phrase "in-place" was
ambiguous *in context*, in either the p5p discussion or the perldelta; it
was only ambiguous in the sense that your specific questions about it
provided no context as to what behaviour was being referring to.

I'm not sure I can parse this sentence correctly. I think I have
provided quite some context. If you think otherwise, feel free to ask.

You asked me some questions about the meaning of 'in-place'. I pointed
out the that phrase could potentially have two meanings, and
I wasn't clear which meaning you were referring to, so I gave answers
for both meanings. I further pointed out that where that phrase had been
used in perldelta / p5p discussion, I thought it was clear from the
context, which meaning was intended there - I wasn't conceding that
perldelta was ambiguous.

--
O Unicef Clearasil!
Gibberish and Drivel!
  -- "Bored of the Rings"

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jul 14, 2018

From @andk

On Sat, 14 Jul 2018 01​:47​:25 -0700, "Dave Mitchell via RT" <perlbug-followup@​perl.org> said​:

  > On Sat, Jul 14, 2018 at 10​:15​:11AM +0200, Andreas Koenig wrote​:

I would argue that

- by having the documentation in the delta manpage,

  > There was no such documentation in perldelta. There was mention of a new
  > optimisation, not of a new sort language feature.

I have cited and llinked to the original places which sentences I speak
about and I maintain that one could read it differently.

  > There is a strong ethos within core development that new optimisations
  > shouldn't change user-visible behaviour where possible. If they do,
  > that's usually considered a bug, or if the trade off is worth it,
  > something to be documented as a gotcha.

It was not noticed for 13 years.

  > Note also that the change was back-ported to 5.8.4 - an odd thing to do
  > for a changed language feature.

I'll cite my timeline again from my previous posting​:

2004-02-20​: https://rt.perl.org/Public/Bug/Display.html?id=25263#txn-79013 Within same RT ticket From​: Dave Mitchell Citation​: "I've now applied change #22349 to bleed, which makes @​a = sort @​a act inplace."

And one line from man perlhist​:

  5.8.4 2004-Apr-21

For my reading, this is not a backport, it was properly injected into
blead between 5.8.3 and 5.8.4.

- not taking it over into the perfunc/sort manpage and

  > There was nothing to take over.

- not noticing the missing part of the documentation for 13 years

  > There was nothing missing.

there is a case of a customary right on the side of all users of the
language to keep the behaviour as implemented.

  > So do you agree that the behaviour which has been present since 5.000 and
  > was accidentally changed in 5.8.4, should be reverted to the 5.000 state,
  > since users have an expectation that the implemented behaviour should be
  > kept?

The changes in RT ticket 25263 got applause for 13 years. I have no
reason to doubt the achievement that came with it. Compare that with the
time between Ilmari's patch and the BBC ticket​: 22 days.

Alternatives have not been suggested yet and might exist that might lead
to an even better solution for all parties.

  > I'm open to suggestions.

First of all I would suggest to document existing behaviour for v5.8.4 -
v5.28 and write tests that make sure we have it covered this time
around. In a second iteration I would hope somebody (hopefully with
decent language designing capabilities) comes up with something that
allows future users to choose which behaviour they want.

PS - I don't think for one minute the use of the phrase "in-place" was
ambiguous *in context*, in either the p5p discussion or the perldelta; it
was only ambiguous in the sense that your specific questions about it
provided no context as to what behaviour was being referring to.

I'm not sure I can parse this sentence correctly. I think I have
provided quite some context. If you think otherwise, feel free to ask.

  > You asked me some questions about the meaning of 'in-place'. I pointed
  > out the that phrase could potentially have two meanings, and
  > I wasn't clear which meaning you were referring to, so I gave answers
  > for both meanings. I further pointed out that where that phrase had been
  > used in perldelta / p5p discussion, I thought it was clear from the
  > context, which meaning was intended there - I wasn't conceding that
  > perldelta was ambiguous.

Acknowledged.

--
andreas

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jul 22, 2018

From @andk

On Sat, 14 Jul 2018 09​:47​:01 +0100, Dave Mitchell <davem@​iabyn.com> said​:

- not taking it over into the perfunc/sort manpage and

  > There was nothing to take over.

- not noticing the missing part of the documentation for 13 years

  > There was nothing missing.

In my previous posting I overlooked this very interesting point in
your argumentation. I think it more than deserves an answer.

perl 5.8.4 introduced (thanks to your effort!) a remarkable user-tunable
optimization on memory usage and you really firmly assert it is not
worth documenting?

Let me present figures how remarkable and how user-tunable it was and
still is (lower is better on all figures)​:

| | max VmPeak |
| perl | @​b=sort@​a | @​a=sort@​a |
|--------+-----------+-----------|
| 5.8.3 | 194524 | 194524 |
| 5.8.4 | 194532 | 114644 |
| 5.28.0 | 171160 | 123768 |

The point is that it is not only user-visible but actually user-tunable.
Your arguments about not having user-visible effects collapse under
these figures. What sense does it make to implement and announce an
optimization that in fact requires cooperation from the user and then
not to document it? And then to claim that it is intended as being
non-user-visible?

--
andreas

PS​: full testrun​:

% for t in `seq 1 3`; do for p in perl-5.8.3/6fdb perl-5.8.4/6fdb v5.28.0/da1c; do
/home/sand/src/perl/repoperls/installed-perls/host/k93msid/$p/bin/perl -le '
my @​a; push @​a, substr(1000000+rand(9000000),0,7) while @​a < 1000000; @​a = sort @​a;
print "$]​: ".join"",`cat /proc/$$/status` =~ /VmPeak​:\s*(\d+)/;
';
/home/sand/src/perl/repoperls/installed-perls/host/k93msid/$p/bin/perl -le '
my @​a; push @​a, substr(1000000+rand(9000000),0,7) while @​a < 1000000; @​b = sort @​a;
print "$]​: ".join"",`cat /proc/$$/status` =~ /VmPeak​:\s*(\d+)/;
';
done; done
5.008003​: 194524
5.008003​: 194524
5.008004​: 114644
5.008004​: 194532
5.028000​: 123768
5.028000​: 171160
5.008003​: 194524
5.008003​: 194524
5.008004​: 114644
5.008004​: 194532
5.028000​: 123768
5.028000​: 171160
5.008003​: 194524
5.008003​: 194524
5.008004​: 114644
5.008004​: 194532
5.028000​: 123768
5.028000​: 171160

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jul 22, 2018

From @iabyn

On Sun, Jul 22, 2018 at 04​:28​:36PM +0200, Andreas Koenig wrote​:

The point is that it is not only user-visible but actually user-tunable.
Your arguments about not having user-visible effects collapse under
these figures. What sense does it make to implement and announce an
optimization that in fact requires cooperation from the user and then
not to document it? And then to claim that it is intended as being
non-user-visible?

Most optimisations I do are "user tunable" by that definition.

Did you know that if you do $a + $b, and ensure that $a and $b have only
ever contained integers (so for example have never held strings or been
used in a string context) then the addition is a lot faster? Is that
documented in perlop? No. It is documented in perldelta? Quite possibly,
in the "Optimizations" section for whatever release I tweaked pp_add() et
al.

I think the position you have taken is extreme, and if we used that
position to inform future perl bug fixes, perl would development would
come to a complete halt.

Really I am utterly confounded by the stance you are taking. Perl has a
strong ethos of maintaining backwards compatibility where reasonable, to
the extent that many many people shout at us for causing the stagnation of
perl 5. But we have never committed to maintaining 100% backwards
compatibility at all costs. Each decision is made on its own merits, with
a big dose of pragmatism.

As I have said before, I am willing to debate the pragmatic pros and cons
of whether it is better to make this change in behaviour. What I am not
prepared to accept, is that digging out a 14 year old RT ticket, which at
no no point proposes that weak references should be preserved, has somehow
bound us in perpetuity to preserve weak references.

--
Indomitable in retreat, invincible in advance, insufferable in victory
  -- Churchill on Montgomery

@p5pRT
Copy link
Collaborator Author

@p5pRT p5pRT commented Jul 29, 2018

From @andk

On Sun, 22 Jul 2018 16​:36​:24 +0100, Dave Mitchell <davem@​iabyn.com> said​:

  > On Sun, Jul 22, 2018 at 04​:28​:36PM +0200, Andreas Koenig wrote​:

The point is that it is not only user-visible but actually user-tunable.
Your arguments about not having user-visible effects collapse under
these figures. What sense does it make to implement and announce an
optimization that in fact requires cooperation from the user and then
not to document it? And then to claim that it is intended as being
non-user-visible?

  > Most optimisations I do are "user tunable" by that definition.

  > Did you know that if you do $a + $b, and ensure that $a and $b have only
  > ever contained integers (so for example have never held strings or been
  > used in a string context) then the addition is a lot faster? Is that
  > documented in perlop? No. It is documented in perldelta? Quite possibly,
  > in the "Optimizations" section for whatever release I tweaked pp_add() et
  > al.

We have talked about it already. Perl is open source and as such the
documentation more often than not is the code itself. This is good
enough when everybody agrees. But when two developers interpret the
source differently, then we must find a balance on how we deal with
those differing interpretations. Then we must discuss what exactly we
consider the desired behaviour. It's one of the basic fundaments of how
open source works and why we engage in it.

  > I think the position you have taken is extreme, and if we used that
  > position to inform future perl bug fixes, perl would development would
  > come to a complete halt.

This is without doubt an interesting question. But watch out, there is a
bit of a grammatical twist in there and I guess it is because you edited
this sentence after having written it. Maybe it indicates, that you are
yourself in doubt about it? In general extremism allegations against
dissenters turn me off.

  > Really I am utterly confounded by the stance you are taking. Perl has a
  > strong ethos of maintaining backwards compatibility where reasonable, to
  > the extent that many many people shout at us for causing the stagnation of
  > perl 5.

Yes, but breaking stuff on CPAN without careful consideration is also
not what we need.

  > But we have never committed to maintaining 100% backwards
  > compatibility at all costs. Each decision is made on its own merits, with
  > a big dose of pragmatism.

Indeed. That's what I'm asking for​: the appropriate amount of
pragmatism. On a case-by-case basis. A careful evaluation, how to
develop the language out of a conflict without brutally removing
valuable features.

  > As I have said before, I am willing to debate the pragmatic pros and cons
  > of whether it is better to make this change in behaviour.

I believe I have made a lot of my points. At least I have digged out the
original RT ticket that was leading to your commit. I find that in-place
sorting without changing any of the elements of an array is a feature
that we should not throw away.

  > What I am not prepared to accept, is that digging out a 14 year old
  > RT ticket, which at no no point proposes that weak references should
  > be preserved, has somehow bound us in perpetuity to preserve weak
  > references.

Your argument is very hard to parse for me, please explain what your
argument is​:

- "digging"​: is it the fact that I was digging in the history what you
  find unacceptable? Are you claiming some right to be forgotten about
  old RT tickets? Or is digging an dirty inappropriate for language
  debate? What else could you have against digging?

- "a 14 year old"​: is it that the discussion is 14 years old? would it
  be more acceptable when the discussion would be more recent? Or older?
  How many years would be OK for you? And justified for which reason?
  Remember, that *my* argument also points out the 14 years. I'm
  arguing, it took 14 years until Tom Molesworth and Ilmari discovered
  this behaviour. For me it is a reason to treat it a feature that just
  has not been documented.

- "at no point proposes that weak references"​: the ticket said
  "in-place" and one reading of "in-place" means​: leave each element
  alone. Leaving each element alone means of course not changing weak
  references. One could argue that the ticket was underspecified. But
  why would that be a justification to reject any different
  interpretation than your own? I know you disagree with my reading but
  then it's simply something we have to agree that we disagree.

- "perpetuity"​: what are you arguing against perpetuity? Has a language
  to be bent and redefined in all its appearances every day to be
  useful? Perpetuity is despite not being realistic, a blessing. For a
  programmer acting as-if it were for the perpetuity should be a
  liability, not a shame.

I repeat my quest​: it is time to document existing behaviour for v5.8.4
- v5.28 and write tests that make sure we have it covered this time
around. In a second iteration I would hope somebody (hopefully with
decent language designing capabilities) comes up with something that
allows future users to choose which behaviour they want.

--
andreas

@toddr
Copy link
Member

@toddr toddr commented Feb 18, 2020

Given this was a BBC ticket and the issue was resolved with a revert of the breaking commit, I don't see any further action required here. I have opened #17569 to decide if we want to put the reverted commit back in the future.

@toddr toddr closed this Feb 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.