suggestion for handling incompatible mandatory features (like experimental warnings)

[p5pRT]

Migrated from rt.perl.org#126314 (status was 'open')

Searchable as RT126314$

[p5pRT]

From perl-diddler@tlinx.org

Created by perl-diddler@tlinx.org

I was installing a new distro package that had some
troublesome new warnings turned on by default as well as
one that changed default UTF-8 handling for the program.

They both served as different examples of how to announce new,
incompatible feature changes (like breaking code with new warnings
where such code, was developed along best practices for most languages
of eliminating all warnings from compile & test, then treating any
new "surprises as fatal".

Both were in the 'ddd-gui' a gui for ddd that drives GDB and DBX with
language specific support. That can show data structures, interactively
as graphs (working with C, C++, Pascal, Modula-2 Fortran, ADA and Assembler.

First problem I noted was a bunch of X errors -- and a suggestion
after the 1st warning was displayed​:

  Warning​: Cannot convert string "-*-verdana-bold-r-*-*-*-120-*-*-*-*-iso8859-*" to type FontStruct
  (Annoyed? Try 'Edit->Preferences->General->Suppress X Warnings'!)
  Warning​: Cannot convert string "-*-verdana-bold-r-*-*-*-100-*-*-*-*-iso8859-*" to type FontStruct
  ....
  8 more font warnings with substitutions...

Sure enough, I went into the gui's "edit-preferences" and clicked it off
and all such errors were suppressed.

In a second case -- a different class of warning caused by an addon-wrapper
script by SUSE that was more prominent w/o the font issues​:

UTF-8 charmap detected. Switching off UTF-8 as ddd has problems with it. See
README.SUSE for futher information. Use -x parameter to bypass this wrapper.

Sure enough... I looked at what used to be a binary for 'ddd' and found
a shell script​:

  # The wrapper script to switch off UTF-8 locales when running ddd
  # See README.SUSE for further information;

The script recognized a "-x" switch to turn off the wrapper effects on
a 1-time basis, and the README said​:

  If you think that warning is annoying, you can turn it off by executing

  $ touch ~/.ddd/suse_no_utf8_warning

  in a shell.
---
  So it also gave a way to permanently suppress the warning about this
new "feature" (auto-disabling of UTF-8), besides 1-time disabling via a
switch. Inherent in the message about the wrapper was implicit information
about how to undo SUSE's disabling (remove the wrapper) and take whatever
bugs may come out of the ddd-gui program.

  Along the lines of both of those solutions for demoting features to
experimental and adding a new non-opt-in feature to display warnings about
the non-opt-in-removal of lexical '$_', as a non-experimental feature
(with no prior notice of the demotion, the new warnings or automatic
error generation for those who learned such "best practices" with
experience in a number of languages (albeit, most, compiled).

Also, don't take my using the specific changes to lexical "$_" as meaning
I felt strongly about them. I'm only using it as an example of
something that was far more likely to cause code problems than, say,
upgrading 'sprintf' to do something useful with an 'array' as printf does,
which, I was told might cause unknown amounts of damage.

Nevertheless, when such incompatibilities arise, since they are
sometimes enabled automatically (vs. the opt-in for new features and
1-cycle (~year) deprecation notice for removals), it seems providing
a safety-net in perl wouldn't be a bad idea.

For the one product I encounted above, they used the existence
of user-choice file-flags, which seems a bit too non-generic
for something with the complexity of perl. So I was thinking
of both system-wide and user-specific (/etc/perl.conf, ~/.perlconfrc)
files that could hold exception information about what new
'default' features to not include, and what old features to not
demote to 'experimental status' such that they trigger the
new new experimental warnings feature (as an example). I don't
think I would recommend it being a place to enable new features,
(ala "use feature "new feature"), as that would make newly
written code possibly incompatible on other people's systems
who didn't have the same settings.

However, making it a way to give the user a way to block
unwanted new behaviors that might cause a work-overload as
many or most perl scripts running the system suddenly
keel over and "die" (literally).

Does that seem reasonable?

Perl Info

Flags:
    category=core
    severity=wishlist

Site configuration information for perl 5.16.3:

Configured by law at Wed Jan 22 12:58:58 PST 2014.

Summary of my perl5 (revision 5 version 16 subversion 3) configuration:
   
  Platform:
    osname=linux, osvers=3.12.0-isht-van, archname=x86_64-linux-thread-multi-ld
    uname='linux ishtar 3.12.0-isht-van #1 smp preempt wed nov 13 16:50:51 pst 2013 x86_64 x86_64 x86_64 gnulinux '
    config_args=''
    hint=previous, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=define
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-g -O2',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
    ccversion='', gccversion='4.8.1 20130909 [gcc-4_8-branch revision 202388]', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='long double', nvsize=16, Off_t='off_t', lseeksize=8
    alignbytes=16, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags ='-g -fstack-protector -fPIC'
    libpth=/usr/lib64 /lib64
    libs=-lnsl -lndbm -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc -lgdbm_compat
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=/lib/libc-2.18.so, so=so, useshrplib=true, libperl=libperl-5.16.3.so
    gnulibc_version='2.18'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/home/perl/perl-5.16.3/lib/x86_64-linux-thread-multi-ld/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -g -O2 -fstack-protector -fPIC'

Locally applied patches:
    


@INC for perl 5.16.3:
    /home/law/bin/lib
    /home/perl/perl-5.16.3/lib/site/x86_64-linux-thread-multi-ld
    /home/perl/perl-5.16.3/lib/site
    /home/perl/perl-5.16.3/lib/x86_64-linux-thread-multi-ld
    /home/perl/perl-5.16.3/lib
    .


Environment for perl 5.16.3:
    HOME=/home/law
    LANG (unset)
    LANGUAGE (unset)
    LC_COLLATE=C
    LC_CTYPE=en_US.UTF-8
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/law/.plenv/shims:/home/law/.plenv/bin:/sbin:.::/home/law/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/usr/sbin:/etc/local/func_lib:/home/law/lib:/home/law/bin/lib
    PERL5OPT=-Mutf8 -CSA -I/home/law/bin/lib
    PERL_BADLANG (unset)
    SHELL=/bin/bash

[p5pRT]

From zefram@fysh.org

There is already a per-host configuration file. However, use of any such
configuration to set warning flags and the like would pose a problem for
programmers, by making Perl a less consistent platform. It is a bad idea.
This ticket should be closed.

-zefram

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @eserte

Dana Thu, 14 Dec 2017 22​:09​:54 -0800, zefram@​fysh.org reče​:

There is already a per-host configuration file. However, use of any such
configuration to set warning flags and the like would pose a problem for
programmers, by making Perl a less consistent platform. It is a bad idea.
This ticket should be closed.

What about an alternative suggestion for the requestor's problems?

Regards,
  Slaven

[p5pRT]

From zefram@fysh.org

slaven@​rezic.de via RT wrote​:

What about an alternative suggestion for the requestor's problems?

The original post wasn't coherent enough for me to discern a specific
problem, only the suggestion promised by the title.

-zefram

[p5pRT]

From @eserte

Dana Thu, 14 Dec 2017 22​:54​:24 -0800, zefram@​fysh.org reče​:

slaven@​rezic.de via RT wrote​:

What about an alternative suggestion for the requestor's problems?

The original post wasn't coherent enough for me to discern a specific
problem, only the suggestion promised by the title.

It is a general problem​: how users should deal with incompatibilities of newer perl versions? What's p5p's suggestion? Stick to older perls (e.g. via perlbrew)? /etc/perl/sitecustomize.pl exists on some platforms (e.g. Debian, or the feature may be added to a self-compiled perl), but should it really be used to mask incompatibilities, and how?

Regards,
  Slaven

[p5pRT]

From zefram@fysh.org

slaven@​rezic.de via RT wrote​:

It is a general problem​: how users should deal with incompatibilities
of newer perl versions?

We limit incompatibilities, and document those that arise in perldelta.
This is a managed problem, and this ticket is not shedding any new light
on it.

-zefram

[p5pRT]

From @eserte

Dana Thu, 14 Dec 2017 23​:21​:47 -0800, zefram@​fysh.org reče​:

slaven@​rezic.de via RT wrote​:

It is a general problem​: how users should deal with incompatibilities
of newer perl versions?

We limit incompatibilities,

I am not sure. I read "we should deprecate it" too often on the mailing list.

and document those that arise in perldelta.

This is just documentation, but not a solution for users (not authors!) encountering such incompatibilities.

Regards,
  Slaven

[p5pRT]

From @xsawyerx

On 12/15/2017 09​:34 AM, slaven@​rezic.de via RT wrote​:

Dana Thu, 14 Dec 2017 23​:21​:47 -0800, zefram@​fysh.org reče​:

slaven@​rezic.de via RT wrote​:

It is a general problem​: how users should deal with incompatibilities
of newer perl versions?
We limit incompatibilities,
I am not sure. I read "we should deprecate it" too often on the mailing list.

I'm a bit wary of making any statement here since some people seem to
monitor the list and quote sentences out of context outside p5p, so I'll
try to be delicate.

There is never value in deprecating in and of itself. It is always a
means to an end - a different end. No one wants to deprecate only as
deprecation. It isn't fun. It is always something we *need* to do,
rather than *want* to do.

We normally deprecate when we need to remove incorrect behavior and bugs
(though often we just keep it in, making the language more compatible
over time but arguably worse - buggy and less capable, as well as less
performant.) I'm well aware that this inconveniences people (myself
included) but in our efforts to have a capable, less buggy language, it
is absolutely critical that we are allowed to make changes. Deprecation
allows us to make these changes over time.

The rise in seeing "we should deprecate it" is likely due to the fact
that p5p is now more active in making a decision on finally removing
some bugs as well as major performance updates. Our exercise of standing
by declared deprecation by actually deprecating following the first p5p
hackathon last year was also a contributing factor to this trend.

I would rather view these as "There is so much in the way of so much."
Deprecation of some of it means allowing us to move forward on some
fronts. This includes new features but also includes cleaning up bugs
and incorrect behavior.

The only possible answers to "I don't like any deprecations" are either
1. Let's leave all the bugs in and get no new features or 2. Let's
change it without giving people time to change their code. Both are
unsatisfactory for a language that wants to move forward rather than
continue running old, featureless, buggy code. It stings, but this
really is the gist of it.

I want to stress that this should always be countered with the price of
a change. Some things will stay buggy and wrong, and there are some
things we will not be able to change or add. It's not an all of nothing,
though some people want all and some people want nothing.

and document those that arise in perldelta.
This is just documentation, but not a solution for users (not authors!) encountering such incompatibilities.

Perhaps a question here is "Is there a way we could raise these warnings
for developers instead of for the users"?

[p5pRT]

From @eserte

Sawyer X <xsawyerx@​gmail.com> writes​:

On 12/15/2017 09​:34 AM, slaven@​rezic.de via RT wrote​:

Dana Thu, 14 Dec 2017 23​:21​:47 -0800, zefram@​fysh.org reče​:

slaven@​rezic.de via RT wrote​:

It is a general problem​: how users should deal with incompatibilities
of newer perl versions?
We limit incompatibilities,
I am not sure. I read "we should deprecate it" too often on the mailing list.

I'm a bit wary of making any statement here since some people seem to
monitor the list and quote sentences out of context outside p5p, so I'll
try to be delicate.

There is never value in deprecating in and of itself. It is always a
means to an end - a different end. No one wants to deprecate only as
deprecation. It isn't fun. It is always something we *need* to do,
rather than *want* to do.

I started to read p5p more closely recently and was mildly shocked about
the amount of comments where breakage of backward compatibility was
proposed. And I had the impression that often it was just for reasons
that things were found too obscure, rarely used and similar.

We normally deprecate when we need to remove incorrect behavior and bugs
(though often we just keep it in, making the language more compatible
over time but arguably worse - buggy and less capable, as well as less
performant.) I'm well aware that this inconveniences people (myself
included) but in our efforts to have a capable, less buggy language, it
is absolutely critical that we are allowed to make changes. Deprecation
allows us to make these changes over time.

The rise in seeing "we should deprecate it" is likely due to the fact
that p5p is now more active in making a decision on finally removing
some bugs as well as major performance updates. Our exercise of standing
by declared deprecation by actually deprecating following the first p5p
hackathon last year was also a contributing factor to this trend.

My fear of a defined deprecation process is that now deprecations will
happen more lightly and more often.

I would rather view these as "There is so much in the way of so much."
Deprecation of some of it means allowing us to move forward on some
fronts. This includes new features but also includes cleaning up bugs
and incorrect behavior.

The only possible answers to "I don't like any deprecations" are either
1. Let's leave all the bugs in and get no new features or 2. Let's
change it without giving people time to change their code. Both are
unsatisfactory for a language that wants to move forward rather than
continue running old, featureless, buggy code. It stings, but this
really is the gist of it.

It would be interesting how many introduced backward incompatibilities
in the past (say, starting from perl 5.10) really had performance
improvements as a result (and are still in place and not reverted
again), and how many new features were really developed thanks to these
changes and currently not marked anymore as experimental (and even still
exist!). Do we have a review process for this? And if the outcome in the
past was not that positive​: will this change in future?

I want to stress that this should always be countered with the price of
a change. Some things will stay buggy and wrong, and there are some
things we will not be able to change or add. It's not an all of nothing,
though some people want all and some people want nothing.

Often such changes are sold as some kind of "progress", be it for
marketing or real reasons. The reality could be quite the opposite ---
developers' work time is wasted for fixing the breakage (instead of
developing new code). Or even worse, authors do nothing. Every new perl
release is a graveyard for a number of CPAN modules, and we don't know
about other Perl code in the wild.

For me as an author this means I have to think hard about really using
CPAN modules as dependencies in my projects --- are there high enough in
the CPAN river and thus unlikely to be broken? And even I have to think
about what Perl features to use --- should I stick to babyperl or may I
still use advanced features?

and document those that arise in perldelta.
This is just documentation, but not a solution for users (not
authors!) encountering such incompatibilities.

Perhaps a question here is "Is there a way we could raise these warnings
for developers instead of for the users"?

I don't see a practical way to distinguish developers from users.

Regards,
  Slaven

--
Slaven Rezic - slaven <at> rezic <dot> de

  Berlin Perl Mongers - http​://berlin.pm.org

[p5pRT]

From perl-diddler@tlinx.org

slaven@​rezic.de via RT wrote​:

Dana Thu, 14 Dec 2017 22​:54​:24 -0800, zefram@​fysh.org reče​:

slaven@​rezic.de via RT wrote​:

What about an alternative suggestion for the requestor's problems?

The original post wasn't coherent enough for me to discern a specific
problem, only the suggestion promised by the title.

It is a general problem​: how users should deal with incompatibilities of newer perl versions? What's p5p's suggestion? Stick to older perls?

Indeed, my _system_'s perl is currently at 5.16.3, since moving higher
triggered
warnings that were regarded as fatal as per strict development practice in
other languages.

[p5pRT]

From @cpansprout

On Sun, 17 Dec 2017 14​:00​:28 -0800, slaven@​rezic.de wrote​:

My fear of a defined deprecation process is that now deprecations will
happen more lightly and more often.

Hear, hear!

--

Father Chrysostomos

[p5pRT]

From @xsawyerx

On 12/17/2017 11​:59 PM, Slaven Rezic wrote​:

Sawyer X <xsawyerx@​gmail.com> writes​:

On 12/15/2017 09​:34 AM, slaven@​rezic.de via RT wrote​:

Dana Thu, 14 Dec 2017 23​:21​:47 -0800, zefram@​fysh.org reče​:

slaven@​rezic.de via RT wrote​:

It is a general problem​: how users should deal with incompatibilities
of newer perl versions?
We limit incompatibilities,
I am not sure. I read "we should deprecate it" too often on the mailing list.
I'm a bit wary of making any statement here since some people seem to
monitor the list and quote sentences out of context outside p5p, so I'll
try to be delicate.

There is never value in deprecating in and of itself. It is always a
means to an end - a different end. No one wants to deprecate only as
deprecation. It isn't fun. It is always something we *need* to do,
rather than *want* to do.
I started to read p5p more closely recently and was mildly shocked about
the amount of comments where breakage of backward compatibility was
proposed. And I had the impression that often it was just for reasons
that things were found too obscure, rarely used and similar.

Those are always open for review. If you see a deprecation that has a
reason too obscure, please surface it for discussion. As I tried
expressing, no one is looking to deprecate. That is *never* a goal. It
is always something that someone thinks needs to be done to accomplish
something else. If you think they are wrong, this is an open forum for
you to say that. (While each objection is heard, not all objections are
accepted.)

If you can prove that the price to accomplish what we want is too great
or that we do not need to deprecate anything to accomplish it, this is
good enough to prevent deprecating. I cannot stress enough, again and
again, that no one enjoys deprecating and we would be happier if we
didn't need to deprecate *anything*.

A common objection to deprecating is the position that we should not
change a thing. That's not an acceptable position. Another unacceptable
position (which seems to be fairly common) is "This doesn't affect me or
my code, so I don't care." Security often hits that problem. Why make a
change to security for users who are not yourself? Easy. The language
serves them as well.

An example of how problematic this is was the change of dot-in-inc. We
were not the only ones to remove current directory from includes, but
when we decided to remove it, many of the calls to keep it (though not
all) were people who did not put any weight on the security-wise harm it
could have to others. I've had emails with authors who simply said "I
understand this is possibly a major security problem, but I just don't
care because it doesn't affect me." That doesn't cut it.

The language must serve multiple people and while it is wonderful if
it's able to run a module someone wrote 20 years ago, it isn't the sole
goal of this language and that module author (or the module users) are
not the only users of said language.

We normally deprecate when we need to remove incorrect behavior and bugs
(though often we just keep it in, making the language more compatible
over time but arguably worse - buggy and less capable, as well as less
performant.) I'm well aware that this inconveniences people (myself
included) but in our efforts to have a capable, less buggy language, it
is absolutely critical that we are allowed to make changes. Deprecation
allows us to make these changes over time.

The rise in seeing "we should deprecate it" is likely due to the fact
that p5p is now more active in making a decision on finally removing
some bugs as well as major performance updates. Our exercise of standing
by declared deprecation by actually deprecating following the first p5p
hackathon last year was also a contributing factor to this trend.
My fear of a defined deprecation process is that now deprecations will
happen more lightly and more often.

This is quite unfair. We cannot argue against not having a deprecation
process and then argue against having one. We cannot eat our cake and
have it too. Are you advocating not having a deprecation process at all?

Having no deprecation process would mean we simply decide something is
deprecated and make it so, such as behaviors who were "deprecated" for
over 20 years. This means a wrong behavior (or a major bug) is left in
place without any plans of actually removing it. That means more people
will depend on it over time. It also means that without a proper
process, we could theoretically remove it at will. You lose on both ends.

Having a proper deprecation process means we at least stand by a
deprecation cycle, which provides a higher standard on what to
deprecate, since deprecating is a process. Additionally, it provides
time to remove and notifies the user by when the code will be removed.

I would rather view these as "There is so much in the way of so much."
Deprecation of some of it means allowing us to move forward on some
fronts. This includes new features but also includes cleaning up bugs
and incorrect behavior.

The only possible answers to "I don't like any deprecations" are either
1. Let's leave all the bugs in and get no new features or 2. Let's
change it without giving people time to change their code. Both are
unsatisfactory for a language that wants to move forward rather than
continue running old, featureless, buggy code. It stings, but this
really is the gist of it.
It would be interesting how many introduced backward incompatibilities
in the past (say, starting from perl 5.10) really had performance
improvements as a result (and are still in place and not reverted
again), and how many new features were really developed thanks to these
changes and currently not marked anymore as experimental (and even still
exist!). Do we have a review process for this? And if the outcome in the
past was not that positive​: will this change in future?

I'm not aware of any statistics on this.

I want to stress that this should always be countered with the price of
a change. Some things will stay buggy and wrong, and there are some
things we will not be able to change or add. It's not an all of nothing,
though some people want all and some people want nothing.
Often such changes are sold as some kind of "progress", be it for
marketing or real reasons. The reality could be quite the opposite ---
developers' work time is wasted for fixing the breakage (instead of
developing new code). Or even worse, authors do nothing. Every new perl
release is a graveyard for a number of CPAN modules, and we don't know
about other Perl code in the wild.

I beg to differ. CPAN itself is a dumping ground for code. We cannot -
and should not - expect anything written to work ad infinitum. That is
not a strong argument.

Can you please provide me with one breaking change that had no reason to it?

Furthermore, just the fact that the term "deprecation" is seen more
negates *nothing* of the agreed-upon reasons for deprecations that p5p
holds. In fact, we stood by deprecation far more recently by finally
following-up on what had been declared deprecated for far too long. So,
yes, you are seeing this word more. Then the occurrences will likely go
down once we are done with the old deprecations. Then it might go up
again. If we find a major security issue, we will likely still want to
fix it, despite possible breakage.

For me as an author this means I have to think hard about really using
CPAN modules as dependencies in my projects --- are there high enough in
the CPAN river and thus unlikely to be broken? And even I have to think
about what Perl features to use --- should I stick to babyperl or may I
still use advanced features?

As an author, you should make sure your code works. Especially if you
use any internal mechanisms of the language and sometimes even if not.
Unfortunately, many modules depend on non-declared and non-intended
behavior. Just on this list we have recently seen an argument of a
certain behavior as feature until the person who wrote it replied with
"I wrote it. It wasn't a feature."

A different situation though​: Let's say you wrote something that used a
common, published mechanism. Next year we find out this allows someone
to delete an entire hard drive of a user once every 3K uses. Would it
make sense updating your module? I would say yes. The dot-in-@​inc is a
good example of that principle. This is a much stronger case for users
of your code than for you.

and document those that arise in perldelta.
This is just documentation, but not a solution for users (not
authors!) encountering such incompatibilities.
Perhaps a question here is "Is there a way we could raise these warnings
for developers instead of for the users"?
I don't see a practical way to distinguish developers from users.

Think of environments. You are developing something using a tool. The
tool might break when you run your tests or in a test environment. This
has little repercussions because that's what those environments are for.
If you are now moving it to production and it breaks - that's bad. When
you're developing an IDE, it's good to see any warnings from your IDE.
But if it's a user using the IDE (where "IDE" could also just be a
module), the warning of the IDE tells them nothing. (IDE was an example
in the original ticket.)

In closing, "I'm seeing the word 'deprecation' more" is not a convincing
argument for us to either stop following-up on declared deprecations or
to fix critical bugs. If you can spot a single deprecation that is
unwarranted, we could happily discuss it. "Deprecations" as a whole are
not debatable because they will happen. The questions are which, why,
and how?