Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

open(): remove 'U' mode, deprecated since Python 3.3 #81511

Closed
vstinner opened this issue Jun 18, 2019 · 25 comments
Closed

open(): remove 'U' mode, deprecated since Python 3.3 #81511

vstinner opened this issue Jun 18, 2019 · 25 comments
Labels
3.11 stdlib Python modules in the Lib dir

Comments

@vstinner
Copy link
Member

vstinner commented Jun 18, 2019

BPO 37330
Nosy @pitrou, @vstinner, @merwok, @ambv, @serhiy-storchaka, @asottile, @miss-islington, @abartlet
PRs
  • bpo-37330: open() no longer accept 'U' in file mode #14204
  • bpo-37330: open() no longer accept 'U' in file mode #16959
  • [3.8] bpo-37330: open(): "U" mode is removed in Python 3.9 #16972
  • [3.7] bpo-37330: open(): "U" mode is removed in Python 3.9 (GH-16972) #16982
  • bpo-39674: Revert "bpo-37330: open() no longer accept 'U' in file mode (GH-16959)" #18767
  • bpo-37330: open() no longer accept 'U' in file mode #28118
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-09-02.11:16:54.115>
    created_at = <Date 2019-06-18.15:48:25.669>
    labels = ['library', '3.11']
    title = "open(): remove 'U' mode, deprecated since Python 3.3"
    updated_at = <Date 2022-01-05.17:46:12.863>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2022-01-05.17:46:12.863>
    actor = 'eric.araujo'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-09-02.11:16:54.115>
    closer = 'vstinner'
    components = ['Library (Lib)']
    creation = <Date 2019-06-18.15:48:25.669>
    creator = 'vstinner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 37330
    keywords = ['patch']
    message_count = 25.0
    messages = ['345991', '345995', '346219', '348007', '355479', '355543', '355544', '355573', '355633', '355635', '355636', '362362', '363194', '363211', '363247', '363320', '363321', '363344', '363366', '400884', '400888', '400889', '400914', '400917', '409791']
    nosy_count = 8.0
    nosy_names = ['pitrou', 'vstinner', 'eric.araujo', 'lukasz.langa', 'serhiy.storchaka', 'Anthony Sottile', 'miss-islington', 'abartlet']
    pr_nums = ['14204', '16959', '16972', '16982', '18767', '28118']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue37330'
    versions = ['Python 3.11']

    @vstinner
    Copy link
    Member Author

    vstinner commented Jun 18, 2019

    The builtin open() function accepts 'U' in the file mode, but it emits a DeprecationWarning since Python 3.4, change made in bpo-15204:

    commit 6787a38
    Author: Serhiy Storchaka <storchaka@gmail.com>
    Date: Sat Nov 23 22:12:06 2013 +0200

    Issue bpo-15204: Deprecated the 'U' mode in file-like objects.
    
    $ cat x.py 
    f = open(__file__, 'U')
    line = f.readline()
    print(type(line))
    f.close()
    
    $ python3.7 x.py 
    x.py:1: DeprecationWarning: 'U' mode is deprecated
      f = open(__file__, 'U')
    <class 'str'>

    The 'U' mode is documented as deprecated since Python 3.3

    The flag is basically only there for backward compatibility with Python 2. Python 3.9.0 final release is currently scheduled at 2020-06-08: 6 months after Python 2 end-of-life (2020-01-01).

    Currently, the 'U' mode is documented as it will be removed from Python 4.0. But I dislike having a bunch of backward incompatible changes in Python 4.0. I would prefer Python 4.0 to be simply the version after Python 3.X.

    Deprecation warnings are now displayed in the __main__ module by default, thanks to the PEP-565 -- Show DeprecationWarning in __main__.
    https://www.python.org/dev/peps/pep-0565/

    More and more projects run their test suite using -Werror and so have to fix deprecation warnings.

    @vstinner vstinner added 3.9 stdlib Python modules in the Lib dir labels Jun 18, 2019
    @vstinner
    Copy link
    Member Author

    vstinner commented Jun 18, 2019

    I prefer to push backward incompatible changes at the beginning of a dev cycle (Python 3.9), so we have room to revert it later if it causes too much trouble.

    @vstinner
    Copy link
    Member Author

    vstinner commented Jun 21, 2019

    I reported the issue to docutils. In fact, docutils FileInput class was already fixed to prevent the deprecation warning, but there was no release yet:
    https://sourceforge.net/p/docutils/bugs/363/

    @vstinner
    Copy link
    Member Author

    vstinner commented Jul 16, 2019

    docutils is already fixed in the development version. I requested a release:
    https://sourceforge.net/p/docutils/bugs/364/

    @vstinner
    Copy link
    Member Author

    vstinner commented Oct 27, 2019

    I cited this change in the PEP-608 "Coordinated Python release".

    @vstinner
    Copy link
    Member Author

    vstinner commented Oct 28, 2019

    New changeset e471e72 by Victor Stinner in branch 'master':
    bpo-37330: open() no longer accept 'U' in file mode (GH-16959)
    e471e72

    @vstinner
    Copy link
    Member Author

    vstinner commented Oct 28, 2019

    Thanks for the review Serhiy ;-)

    @asottile
    Copy link
    Mannequin

    asottile mannequin commented Oct 28, 2019

    should we backport a documentation change for this? (the deprecatedremoved says 4.0 currently)

    @vstinner
    Copy link
    Member Author

    vstinner commented Oct 29, 2019

    New changeset 1d2862a by Victor Stinner in branch '3.8':
    bpo-37330: open(): "U" mode is removed in Python 3.9 (GH-16972)
    1d2862a

    @miss-islington
    Copy link
    Contributor

    miss-islington commented Oct 29, 2019

    New changeset 4cb08b6 by Miss Skeleton (bot) in branch '3.7':
    bpo-37330: open(): "U" mode is removed in Python 3.9 (GH-16972)
    4cb08b6

    @vstinner
    Copy link
    Member Author

    vstinner commented Oct 29, 2019

    should we backport a documentation change for this? (the deprecatedremoved says 4.0 currently)

    Done. Thanks for the reminder.

    @abartlet
    Copy link
    Mannequin

    abartlet mannequin commented Feb 20, 2020

    This breaks Samba's build:

    https://bugzilla.samba.org/show_bug.cgi?id=14266

    While we are of course able to patch new versions of Samba, this will make it harder to bisect back though Samba history. It would be really great if this kind of change could be reverted or at least deferred so Samba can rely on a stable language platform for it's build system.

    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 2, 2020

    While we are of course able to patch new versions of Samba, this will make it harder to bisect back though Samba history.

    I don't understand the bisect part. Would you mind to elaborate? What do you want to bisect?

    It would be really great if this kind of change could be reverted or at least deferred so Samba can rely on a stable language platform for it's build system.

    The "U" mode was deprecated since September 2012 (Python 3.3): you had 8 years to handle the DeprecationWarning. 8 years before actually removing a deprecated feature sounds like a stable language platform to me.

    I'm not sure of the benefit of deferring the change. Does it mean that Sambda would not be updated?

    Well, your comment fits exactly into bpo-39674 scope, except that revert the "U" mode is not scheduled yet.

    @abartlet
    Copy link
    Mannequin

    abartlet mannequin commented Mar 2, 2020

    Thanks for the reply.

    To clarify, it is best to understand where Samba thinks of itself in terms of Python. We use python, but we don't really think of ourselves as a python project, and while we have some developers with a depth of experience in the language, most of us just use it as a tool to get things done, in particular to build Samba.

    Samba has a build system built in Python, called Waf. Samba uses waf to build the project (particularly the C parts), and our build of the C binaries is portable back to Python 2.6.

    Often in Samba we find bugs. This might come as a shock! ;-). Many of those bugs are regressions, and so it is useful to build old versions, sometimes 7 year old versions, to confirm if/when the behaviour regressed. We are able to do this, with only one patch to address a perl behaviour change, even on a modern Ubuntu 18.04 system, all the way back to 2013.

    I terms of 'but this has been deprecated for years', Samba has only had releases been able to even use Python3 in the build system for a year (first committed to master in July 2018).

    Nobody in Samba was aware of any DeprecationWarning and for better or worse we only knew about the issue once our builds failed in Fedora. We are incredibly grateful that Fedora tried a rebuild with the alpha version of Python 3.9 as otherwise we would find out only after this was well baked into a release.

    Thankfully we don't need universal newlines, but no matter if we need them, old versions of Samba will fail to build without a fix-up patch.

    More importantly, I think it is unlikely that Samba is the only software to have used this feature, intentionally or otherwise, or to be in this position.

    While not as critical here, for context as to why we support Python 2.6 to build, it should be noted that retaining this compatibility was a critical part of being able to get consensus to otherwise move to Python 3.4.

    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 3, 2020

    Thankfully we don't need universal newlines, but no matter if we need them, old versions of Samba will fail to build without a fix-up patch.

    io.open() is available since Python 2.6 and handles different kinds of newlines transparently. Example:

    $ python2
    Python 2.7.17 (default, Oct 20 2019, 00:00:00) 
    >>> f=open("test", "wb"); f.write("a\rb\rc\r"); f.close()
    >>> import io; f=io.open("test"); lines=list(f); f.close(); lines
    [u'a\n', u'b\n', u'c\n']

    Do you suggest to change the documentation to suggest to open io.open() for applications which still care about Python 2?
    https://docs.python.org/dev/whatsnew/3.9.html#changes-in-the-python-api

    Note: Python 2 is no longer supported upstream, I strongly advice you to upgrade to Python 3.

    @abartlet
    Copy link
    Mannequin

    abartlet mannequin commented Mar 4, 2020

    I think we are speaking past each other.

    Yes, Python 2 is no longer being changed, which is awesome as we need for fear breakage of the older builds that use that for the build system.

    The issue isn't the particular language feature, or that there is no way to further adapt code, but that a valid build script that operates correctly on Python 3.8 will no longer, without warning to us, operate on Python 2.9 or at best 2.10.

    Thankfully the only Samba versions with support for Python3 are still in maintenance, so course I've back-ported patches to the tip of each release branch to work around this change. But again, that doesn't help when I need to build an arbitrary git revision in the future.

    (That warnings were available, if we knew were to look, is an entirely moot point at this stage).

    @abartlet
    Copy link
    Mannequin

    abartlet mannequin commented Mar 4, 2020

    ...Python 2.9 or at best 2.10.

    Of course I mean Python 3.9 or at best 3.10.

    Sorry for the confusion.

    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 4, 2020

    Le mer. 4 mars 2020 à 03:01, Andrew Bartlett <report@bugs.python.org> a écrit :

    (...) but that a valid build script that operates correctly on Python 3.8 will no longer, without warning to us, (...)

    Displaying warnings by default or not is an old debate.

    I added the "Python Development Mode" (-X dev) to help the situation:

    https://docs.python.org/dev/library/devmode.html#devmode

    I just added a very explicit "You should check for DeprecationWarning in your code" section in What's New In Python 3.9:

    https://docs.python.org/dev/whatsnew/3.9.html#you-should-check-for-deprecationwarning-in-your-code

    Moreover, DeprecationWarning are now (Python 3.7) displayed *by default* in the __main__ module:
    https://www.python.org/dev/peps/pep-0565/

    @vstinner
    Copy link
    Member Author

    vstinner commented Mar 4, 2020

    New changeset 942f7a2 by Victor Stinner in branch 'master':
    bpo-39674: Revert "bpo-37330: open() no longer accept 'U' in file mode (GH-16959)" (GH-18767)
    942f7a2

    @vstinner
    Copy link
    Member Author

    vstinner commented Sep 1, 2021

    I reopen the issue. I created PR 28118 to try again to remove the "U" mode in Python 3.11.

    @vstinner vstinner added 3.11 and removed 3.9 labels Sep 1, 2021
    @vstinner vstinner reopened this Sep 1, 2021
    @vstinner
    Copy link
    Member Author

    vstinner commented Sep 2, 2021

    This breaks Samba's build:
    https://bugzilla.samba.org/show_bug.cgi?id=14266

    Samba has been updated in the meanwhile, buildtools/wafsamba/samba_utils.py:

    •            function_code = node.read('rU', None)
      

    + function_code = node.read('r', None)

    https://github.com/samba-team/samba/blob/1209c89dcf6371bbfa4f3929a47a573ef2916c1a/buildtools/wafsamba/samba_utils.py#L692

    @vstinner
    Copy link
    Member Author

    vstinner commented Sep 2, 2021

    docutils is already fixed in the development version. I requested a release: https://sourceforge.net/p/docutils/bugs/364/

    At 2019-07-22, Günter Milde wrote: "Docutils 0.15 is released" (with the fix). The latest docutils version is 0.17.1.

    @ambv
    Copy link
    Contributor

    ambv commented Sep 2, 2021

    New changeset 19ba212 by Victor Stinner in branch 'main':
    bpo-37330: open() no longer accept 'U' in file mode (GH-28118)
    19ba212

    @vstinner
    Copy link
    Member Author

    vstinner commented Sep 2, 2021

    Let's see how it goes. In the worst case, we have time before Python 3.11 final to revert it one more time.

    @vstinner vstinner closed this as completed Sep 2, 2021
    @merwok
    Copy link
    Member

    merwok commented Jan 5, 2022

    it is useful to build old versions, sometimes 7 year old versions, to confirm if/when the behaviour regressed.

    I think the sticking point is this: it is not expected that old versions should work as-is with the newest Python. Can you do your build and test with the Python version that was current 7 years ago?

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.11 stdlib Python modules in the Lib dir
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants