Navigation Menu

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove old-deprecated ElementTree features (part 2) #80724

Closed
serhiy-storchaka opened this issue Apr 6, 2019 · 27 comments
Closed

Remove old-deprecated ElementTree features (part 2) #80724

serhiy-storchaka opened this issue Apr 6, 2019 · 27 comments
Assignees
Labels
3.9 only security fixes topic-XML type-feature A feature request or enhancement

Comments

@serhiy-storchaka
Copy link
Member

BPO 36543
Nosy @scoder, @tiran, @encukou, @serhiy-storchaka, @hroncok, @miss-islington, @rahul-kumi
PRs
  • [3.9] bpo-36543: Remove old-deprecated ElementTree features. #12707
  • bpo-39937: modify iter() suggestion to Element.iter() #18937
  • bpo-36543: Remove the xml.etree.cElementTree module. #19108
  • bpo-36543: What's new: Document how to replace xml.etree.cElementTree #19188
  • Revert "bpo-36543: Remove the xml.etree.cElementTree module." #19920
  • bpo-36543: Restore cElementTree and mark it for removal in 3.10 #19921
  • bpo-36543: Revert "bpo-36543: Remove the xml.etree.cElementTree module." #20117
  • [3.9] bpo-36543: Revert "bpo-36543: Remove the xml.etree.cElementTree module." (GH-20117) #20780
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/serhiy-storchaka'
    closed_at = <Date 2020-06-10.16:04:24.719>
    created_at = <Date 2019-04-06.16:06:22.348>
    labels = ['expert-XML', 'type-feature', '3.9']
    title = 'Remove old-deprecated ElementTree features (part 2)'
    updated_at = <Date 2021-01-05.17:46:41.989>
    user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2021-01-05.17:46:41.989>
    actor = 'serhiy.storchaka'
    assignee = 'serhiy.storchaka'
    closed = True
    closed_date = <Date 2020-06-10.16:04:24.719>
    closer = 'scoder'
    components = ['XML']
    creation = <Date 2019-04-06.16:06:22.348>
    creator = 'serhiy.storchaka'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 36543
    keywords = ['patch']
    message_count = 27.0
    messages = ['339533', '339535', '350938', '364802', '365190', '368083', '368105', '368108', '368111', '368113', '368118', '368120', '368121', '368122', '368123', '368127', '368135', '371210', '371212', '371214', '384390', '384399', '384400', '384401', '384412', '384415', '384419']
    nosy_count = 8.0
    nosy_names = ['scoder', 'christian.heimes', 'eli.bendersky', 'petr.viktorin', 'serhiy.storchaka', 'hroncok', 'miss-islington', 'rahul-kumi']
    pr_nums = ['12707', '18937', '19108', '19188', '19920', '19921', '20117', '20780']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue36543'
    versions = ['Python 3.9']

    @serhiy-storchaka
    Copy link
    Member Author

    The proposed PR removes old-deprecated ElementTree features.

    • Methods Element.getchildren(), Element.getiterator() and ElementTree.getiterator() deprecated in 2.7 and 3.2. They were deprecated in the documentation only, and started to emit a warning in 3.8. Use list(elem) or iteration instead of getchildren(), methods iter() instead of getiterator().

    • The xml.etree.cElementTree module deprecated in 3.3. It was deprecated documentation only because adding a runtime warning will cause more harm than removing it because of the common idiom of using it since Python 2:

      try:
      import xml.etree.cElementTree as ET
      except ImportError:
      import xml.etree.ElementTree as ET

    TODO: Add a What's New entry after the start of developing 3.9.

    @serhiy-storchaka serhiy-storchaka added the 3.9 only security fixes label Apr 6, 2019
    @serhiy-storchaka serhiy-storchaka self-assigned this Apr 6, 2019
    @serhiy-storchaka serhiy-storchaka added topic-XML type-feature A feature request or enhancement labels Apr 6, 2019
    @serhiy-storchaka
    Copy link
    Member Author

    See also bpo-29209.

    @serhiy-storchaka
    Copy link
    Member Author

    New changeset f02ea62 by Serhiy Storchaka in branch 'master':
    bpo-36543: Remove old-deprecated ElementTree features. (GH-12707)
    f02ea62

    @serhiy-storchaka
    Copy link
    Member Author

    New changeset b33e525 by Serhiy Storchaka in branch 'master':
    bpo-36543: Remove the xml.etree.cElementTree module. (GH-19108)
    b33e525

    @serhiy-storchaka
    Copy link
    Member Author

    New changeset 6467134 by Miro Hrončok in branch 'master':
    bpo-36543: What's new: Document how to replace xml.etree.cElementTree (GH-19188)
    6467134

    @tiran
    Copy link
    Member

    tiran commented May 4, 2020

    For the record, I'm quite unhappy that xml.etree.cElementTree was removed without going through a proper active deprecation cycle with plenty of head start. The removal came as a surprise to me -- and I'm a core dev, author of PEP-594 and owner of the defusedxml package. xml.etree.cElementTree module was also never added to PEP-4 and is still not mentioned as removed in PEP-4.

    This changeset broke defusedxml, a package with over 10M downloads a month and 57k dependencies on Github. I'm sure that the removal is going to cause more trouble for other packages, too.

    In PEP-594 I deliberately included a long deprecation phase for all packages with at least one release deprecation warnings. I gave users plenty of time to plan for removal.

    @scoder
    Copy link
    Contributor

    scoder commented May 5, 2020

    Christian, I understand your complaint, but I've actually never seen code in the wild that didn't provide a fallback for the import, usually something like what Serhiy spelled out and explained above:

    try:
        import xml.etree.cElementTree as ET
    except ImportError:
        import xml.etree.ElementTree as ET
    

    This makes it inconvenient for users to emit a deprecation warning for the import, because it would impact perfectly future proof code like the above, without a good way for users to work around it by adapting their code, because the above import order is actually what they want. And there's a lot of such code.

    I personally believe that the breakage will be quite limited. But that's a belief, not a fact. I don't have numbers to back it.

    @serhiy-storchaka
    Copy link
    Member Author

    xml.etree.cElementTree should be treated in the same way as cStringIO and cPickle -- they where separate modules in the past, but now the acceleration is used by default in io.StringIO and pickle. It looks an oversign that it was not removed in 3.0.

    xml.etree.cElementTree was deprecated since 3.3. I think 6 releases (3.3-3.8) is long enough period. It was not possible to emit a deprecation warning at runtime for reasons mentioned above. And we did not have deprecation warnings for cStringIO and cPickle for the same reasons.

    defusedxml is the only known to me package which does not fall back to xml.etree.ElementTree. I hope it is easy to fix it. In Python xml.etree.cElementTree is the same as xml.etree.ElementTree, so you can just import xml.etree.ElementTree. And you can use the above mentioned idiom if you still support Python 2.

    @hroncok
    Copy link
    Mannequin

    hroncok mannequin commented May 5, 2020

    Some stats: See https://bugzilla.redhat.com/showdependencytree.cgi?id=PYTHON39&hide_resolved=0 and Ctrl+F for cElementTree.

    @scoder
    Copy link
    Contributor

    scoder commented May 5, 2020

    Thanks Miro, that's beautiful. Good to know that I'm not the only one who doesn't read documentation. ;-)

    So, how do we deal with this? We can't warn about the deprecation without annoying everyone. We can't just remove the empty module without breaking lots of code. We also can't really let the module stick around forever and let people write more code that uselessly imports it.

    Any ideas?

    @scoder scoder reopened this May 5, 2020
    @serhiy-storchaka
    Copy link
    Member Author

    Thank you Miro for information. It looks as awesome work!

    The simplest solution would be to restore xml.etree.cElementTree and keep it forever. It will not harm anyone, it is just outdated import here and there.

    We could also write an import hook which would analyze the code of the importer and warn only if xml.etree.cElementTree is imported without a common idiom. But it looks slightly overcomplicated to me.

    And we can just let the users to fix they code for 3.9.

    @tiran
    Copy link
    Member

    tiran commented May 5, 2020

    Stefan wrote:

    Christian, I understand your complaint, but I've actually never seen code in the wild that didn't provide a fallback for the import, usually something like what Serhiy spelled out

    My package defusedxml broke with the change. Python's documention recommends my package for parsing XML from untrusted sources. defusedxml 0.7.0rc1 addresses the problem by removing defusedxml.cElementTree in 3.9, too. The fix is going to break other packages, too.

    Serhiy wrote:

    xml.etree.cElementTree should be treated in the same way as cStringIO and cPickle -- they where separate modules in the past, but now the acceleration is used by default in io.StringIO and pickle. It looks an oversign that it was not removed in 3.0.

    I completely agree with you. It should have been removed with 3.0. But this fact doesn't grant a carte blanche to remove the module without a deprecation phase with a proper deadline.

    Stefan wrote:

    So, how do we deal with this? We can't warn about the deprecation without annoying everyone. We can't just remove the empty module without breaking lots of code. We also can't really let the module stick around forever and let people write more code that uselessly imports it.

    I prefer annoyance over getting broking by surprise. If the current deprecation warning system doesn't work for us, then we need to come up with a better plan. In the mean time please give users one release head star to adjust their code. I propose that feature shall not be removed in 3.x unless the documentation of 3.(x-1).0 release has explicitly stated the removal of a feature (unless other special circumstances and general consense require it).

    • Add deprecation and removal warning for xml.etree.cElementTree to whatsnew/3.9.rst
    • Remove xml.etree.cElementTree in 3.10

    @tiran
    Copy link
    Member

    tiran commented May 5, 2020

    IMHO we don't have to spend extra work on an import hook. It might be a good idea to raise the problem with https://github.com/PyCQA/, though. Large projects tend to use code checkers and linters.

    @scoder
    Copy link
    Contributor

    scoder commented May 5, 2020

    It's not like the deprecation was hidden – the ET docs have this line right at the top:

    "Changed in version 3.3: This module will use a fast implementation whenever available. The xml.etree.cElementTree module is deprecated."

    https://docs.python.org/3/library/xml.etree.elementtree.html

    But as I said, people don't read docs.

    I agree that it's best to keep the module "for now", and maybe leave it to IDEs and code checkers to spread the deprecation notice for us. As Serhiy said, it doesn't hurt us much to have it.

    @tiran
    Copy link
    Member

    tiran commented May 5, 2020

    3.3.0 was released in 2012. That's almost a lifetime ago. I assume that majority of users either forgot, missed the note, or no longer took the deprecation serious. After seven, eight years we should do our users a service and point out the deprecation with a concrete deadline.

    @scoder
    Copy link
    Contributor

    scoder commented May 5, 2020

    > But as I said, people don't read docs.

    After seven, eight years we should do our users a service and point out the deprecation with a concrete deadline.

    ¯\(ツ)

    Honestly, let's just keep it. Maybe we can add an invisible PendingDeprecationWarning and wait if some static analysers pick it up. I wouldn't want to invest any more work into it than that.

    @serhiy-storchaka
    Copy link
    Member Author

    I do not think that it is right to emit a warning at import time. We should not punish people which do things right: fallback to xml.etree.ElementTree and test with -We.

    @scoder
    Copy link
    Contributor

    scoder commented Jun 10, 2020

    New changeset ec88e1b by Serhiy Storchaka in branch 'master':
    bpo-36543: Revert "bpo-36543: Remove the xml.etree.cElementTree module." (GH-20117)
    ec88e1b

    @scoder
    Copy link
    Contributor

    scoder commented Jun 10, 2020

    New changeset 3b97d1b by Miss Islington (bot) in branch '3.9':
    bpo-36543: Revert "bpo-36543: Remove the xml.etree.cElementTree module." (GH-20117) (GH-20780)
    3b97d1b

    @scoder
    Copy link
    Contributor

    scoder commented Jun 10, 2020

    The xml.etree.cElementTree module is restored. We'll see about what to do with it later. It hurts less to keep it around than to delete it.

    I think this ticket can be closed.

    @scoder scoder closed this as completed Jun 10, 2020
    @encukou
    Copy link
    Member

    encukou commented Jan 5, 2021

    Let me just point out there's an official process for making incompatible changes now: https://www.python.org/dev/peps/pep-0387/#making-incompatible-changes

    A warning must be emitted (not just mentioned in docs) for two releases before a feature is removed.

    @serhiy-storchaka
    Copy link
    Member Author

    This process is not new. The problem is that in this particular case warning will do more harm than removing the module. If we cannot just remove it, we should not introduce warning. It will be kept forever.

    @encukou
    Copy link
    Member

    encukou commented Jan 5, 2021

    Should the process be changed?

    @tiran
    Copy link
    Member

    tiran commented Jan 5, 2021

    The process is fine. A deprecation warning will give users time to update code before it hits production.

    @serhiy-storchaka
    Copy link
    Member Author

    Deprecation warning may help users of old lower-quality code at the cost of harming users of old higher-quality code. It does not look right to me.

    @tiran
    Copy link
    Member

    tiran commented Jan 5, 2021

    Apparently my defusedxml package is low-quality code...

    Anyhow PEP-387 has clear rules how to handle incompatible changes. Every core dev has to follow these rules. You cannot simply ignore the rule, because you don't like it. If you want to change the rules or get an exception for a special case, please open a discussion, lobby for an amendment to PEP-387, and get approval from the SC.

    @serhiy-storchaka
    Copy link
    Member Author

    I do not understand what this discussion in a long-closed article is about. If you propose to add warning, I am against it. You are against removing the module. Well, the module will not be removed, the warning is no longer needed. The issue is closed.

    I do not propose to change the rule. I say that the procedure cannot be executed in this case, because it will cause more harm than benefit. If we cannot follow the procedure for removing module, we cannot remove that module. It is fine to me. If you want to change this, propose changes to the procedure, but it should be discussed not here.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    nueces added a commit to nueces/jenkins-job-wrecker that referenced this issue Oct 30, 2023
    The getchildren method was deprecated in python2.7 and is removed 3.10
    See:
    - python/cpython#73395
    - python/cpython#80724
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.9 only security fixes topic-XML type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants