Remove old-deprecated ElementTree features (part 2)

[serhiy-storchaka]

BPO	36543
Nosy	@scoder, @tiran, @encukou, @serhiy-storchaka, @hroncok, @miss-islington, @rahul-kumi
PRs	[3.9] bpo-36543: Remove old-deprecated ElementTree features. #12707 bpo-39937: modify iter() suggestion to Element.iter() #18937 bpo-36543: Remove the xml.etree.cElementTree module. #19108 bpo-36543: What's new: Document how to replace xml.etree.cElementTree #19188 Revert "bpo-36543: Remove the xml.etree.cElementTree module." #19920 bpo-36543: Restore cElementTree and mark it for removal in 3.10 #19921 bpo-36543: Revert "bpo-36543: Remove the xml.etree.cElementTree module." #20117 [3.9] bpo-36543: Revert "bpo-36543: Remove the xml.etree.cElementTree module." (GH-20117) #20780

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2020-06-10.16:04:24.719>
created_at = <Date 2019-04-06.16:06:22.348>
labels = ['expert-XML', 'type-feature', '3.9']
title = 'Remove old-deprecated ElementTree features (part 2)'
updated_at = <Date 2021-01-05.17:46:41.989>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2021-01-05.17:46:41.989>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2020-06-10.16:04:24.719>
closer = 'scoder'
components = ['XML']
creation = <Date 2019-04-06.16:06:22.348>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 36543
keywords = ['patch']
message_count = 27.0
messages = ['339533', '339535', '350938', '364802', '365190', '368083', '368105', '368108', '368111', '368113', '368118', '368120', '368121', '368122', '368123', '368127', '368135', '371210', '371212', '371214', '384390', '384399', '384400', '384401', '384412', '384415', '384419']
nosy_count = 8.0
nosy_names = ['scoder', 'christian.heimes', 'eli.bendersky', 'petr.viktorin', 'serhiy.storchaka', 'hroncok', 'miss-islington', 'rahul-kumi']
pr_nums = ['12707', '18937', '19108', '19188', '19920', '19921', '20117', '20780']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue36543'
versions = ['Python 3.9']

[serhiy-storchaka]

The proposed PR removes old-deprecated ElementTree features.

• Methods Element.getchildren(), Element.getiterator() and ElementTree.getiterator() deprecated in 2.7 and 3.2. They were deprecated in the documentation only, and started to emit a warning in 3.8. Use list(elem) or iteration instead of getchildren(), methods iter() instead of getiterator().

• The xml.etree.cElementTree module deprecated in 3.3. It was deprecated documentation only because adding a runtime warning will cause more harm than removing it because of the common idiom of using it since Python 2:

  try:
  import xml.etree.cElementTree as ET
  except ImportError:
  import xml.etree.ElementTree as ET

TODO: Add a What's New entry after the start of developing 3.9.

[serhiy-storchaka]

See also bpo-29209.

[serhiy-storchaka]

New changeset f02ea62 by Serhiy Storchaka in branch 'master':
bpo-36543: Remove old-deprecated ElementTree features. (GH-12707)
f02ea62

[serhiy-storchaka]

New changeset b33e525 by Serhiy Storchaka in branch 'master':
bpo-36543: Remove the xml.etree.cElementTree module. (GH-19108)
b33e525

[serhiy-storchaka]

New changeset 6467134 by Miro Hrončok in branch 'master':
bpo-36543: What's new: Document how to replace xml.etree.cElementTree (GH-19188)
6467134

[tiran]

For the record, I'm quite unhappy that xml.etree.cElementTree was removed without going through a proper active deprecation cycle with plenty of head start. The removal came as a surprise to me -- and I'm a core dev, author of PEP-594 and owner of the defusedxml package. xml.etree.cElementTree module was also never added to PEP-4 and is still not mentioned as removed in PEP-4.

This changeset broke defusedxml, a package with over 10M downloads a month and 57k dependencies on Github. I'm sure that the removal is going to cause more trouble for other packages, too.

In PEP-594 I deliberately included a long deprecation phase for all packages with at least one release deprecation warnings. I gave users plenty of time to plan for removal.

[scoder]

Christian, I understand your complaint, but I've actually never seen code in the wild that didn't provide a fallback for the import, usually something like what Serhiy spelled out and explained above:

try:
    import xml.etree.cElementTree as ET
except ImportError:
    import xml.etree.ElementTree as ET

This makes it inconvenient for users to emit a deprecation warning for the import, because it would impact perfectly future proof code like the above, without a good way for users to work around it by adapting their code, because the above import order is actually what they want. And there's a lot of such code.

I personally believe that the breakage will be quite limited. But that's a belief, not a fact. I don't have numbers to back it.

[serhiy-storchaka]

xml.etree.cElementTree should be treated in the same way as cStringIO and cPickle -- they where separate modules in the past, but now the acceleration is used by default in io.StringIO and pickle. It looks an oversign that it was not removed in 3.0.

xml.etree.cElementTree was deprecated since 3.3. I think 6 releases (3.3-3.8) is long enough period. It was not possible to emit a deprecation warning at runtime for reasons mentioned above. And we did not have deprecation warnings for cStringIO and cPickle for the same reasons.

defusedxml is the only known to me package which does not fall back to xml.etree.ElementTree. I hope it is easy to fix it. In Python xml.etree.cElementTree is the same as xml.etree.ElementTree, so you can just import xml.etree.ElementTree. And you can use the above mentioned idiom if you still support Python 2.

[hroncok]

Some stats: See https://bugzilla.redhat.com/showdependencytree.cgi?id=PYTHON39&hide_resolved=0 and Ctrl+F for cElementTree.

[scoder]

Thanks Miro, that's beautiful. Good to know that I'm not the only one who doesn't read documentation. ;-)

So, how do we deal with this? We can't warn about the deprecation without annoying everyone. We can't just remove the empty module without breaking lots of code. We also can't really let the module stick around forever and let people write more code that uselessly imports it.

Any ideas?

[serhiy-storchaka]

Thank you Miro for information. It looks as awesome work!

The simplest solution would be to restore xml.etree.cElementTree and keep it forever. It will not harm anyone, it is just outdated import here and there.

We could also write an import hook which would analyze the code of the importer and warn only if xml.etree.cElementTree is imported without a common idiom. But it looks slightly overcomplicated to me.

And we can just let the users to fix they code for 3.9.

[tiran]

Stefan wrote:

Christian, I understand your complaint, but I've actually never seen code in the wild that didn't provide a fallback for the import, usually something like what Serhiy spelled out

My package defusedxml broke with the change. Python's documention recommends my package for parsing XML from untrusted sources. defusedxml 0.7.0rc1 addresses the problem by removing defusedxml.cElementTree in 3.9, too. The fix is going to break other packages, too.

Serhiy wrote:

xml.etree.cElementTree should be treated in the same way as cStringIO and cPickle -- they where separate modules in the past, but now the acceleration is used by default in io.StringIO and pickle. It looks an oversign that it was not removed in 3.0.

I completely agree with you. It should have been removed with 3.0. But this fact doesn't grant a carte blanche to remove the module without a deprecation phase with a proper deadline.

Stefan wrote:

So, how do we deal with this? We can't warn about the deprecation without annoying everyone. We can't just remove the empty module without breaking lots of code. We also can't really let the module stick around forever and let people write more code that uselessly imports it.

I prefer annoyance over getting broking by surprise. If the current deprecation warning system doesn't work for us, then we need to come up with a better plan. In the mean time please give users one release head star to adjust their code. I propose that feature shall not be removed in 3.x unless the documentation of 3.(x-1).0 release has explicitly stated the removal of a feature (unless other special circumstances and general consense require it).

• Add deprecation and removal warning for xml.etree.cElementTree to whatsnew/3.9.rst
• Remove xml.etree.cElementTree in 3.10

[tiran]

IMHO we don't have to spend extra work on an import hook. It might be a good idea to raise the problem with https://github.com/PyCQA/, though. Large projects tend to use code checkers and linters.

[scoder]

It's not like the deprecation was hidden – the ET docs have this line right at the top:

"Changed in version 3.3: This module will use a fast implementation whenever available. The xml.etree.cElementTree module is deprecated."

https://docs.python.org/3/library/xml.etree.elementtree.html

But as I said, people don't read docs.

I agree that it's best to keep the module "for now", and maybe leave it to IDEs and code checkers to spread the deprecation notice for us. As Serhiy said, it doesn't hurt us much to have it.

[tiran]

3.3.0 was released in 2012. That's almost a lifetime ago. I assume that majority of users either forgot, missed the note, or no longer took the deprecation serious. After seven, eight years we should do our users a service and point out the deprecation with a concrete deadline.

[scoder]

> But as I said, people don't read docs.

After seven, eight years we should do our users a service and point out the deprecation with a concrete deadline.

¯\(ツ)/¯

Honestly, let's just keep it. Maybe we can add an invisible PendingDeprecationWarning and wait if some static analysers pick it up. I wouldn't want to invest any more work into it than that.

[serhiy-storchaka]

I do not think that it is right to emit a warning at import time. We should not punish people which do things right: fallback to xml.etree.ElementTree and test with -We.

[scoder]

New changeset ec88e1b by Serhiy Storchaka in branch 'master':
bpo-36543: Revert "bpo-36543: Remove the xml.etree.cElementTree module." (GH-20117)
ec88e1b

[scoder]

New changeset 3b97d1b by Miss Islington (bot) in branch '3.9':
bpo-36543: Revert "bpo-36543: Remove the xml.etree.cElementTree module." (GH-20117) (GH-20780)
3b97d1b

[scoder]

The xml.etree.cElementTree module is restored. We'll see about what to do with it later. It hurts less to keep it around than to delete it.

I think this ticket can be closed.

[encukou]

Let me just point out there's an official process for making incompatible changes now: https://www.python.org/dev/peps/pep-0387/#making-incompatible-changes

A warning must be emitted (not just mentioned in docs) for two releases before a feature is removed.

[serhiy-storchaka]

This process is not new. The problem is that in this particular case warning will do more harm than removing the module. If we cannot just remove it, we should not introduce warning. It will be kept forever.

[encukou]

Should the process be changed?

[tiran]

The process is fine. A deprecation warning will give users time to update code before it hits production.

[serhiy-storchaka]

Deprecation warning may help users of old lower-quality code at the cost of harming users of old higher-quality code. It does not look right to me.

[tiran]

Apparently my defusedxml package is low-quality code...

Anyhow PEP-387 has clear rules how to handle incompatible changes. Every core dev has to follow these rules. You cannot simply ignore the rule, because you don't like it. If you want to change the rules or get an exception for a special case, please open a discussion, lobby for an amendment to PEP-387, and get approval from the SC.

[serhiy-storchaka]

I do not understand what this discussion in a long-closed article is about. If you propose to add warning, I am against it. You are against removing the module. Well, the module will not be removed, the warning is no longer needed. The issue is closed.

I do not propose to change the rule. I say that the procedure cannot be executed in this case, because it will cause more harm than benefit. If we cannot follow the procedure for removing module, we cannot remove that module. It is fine to me. If you want to change this, propose changes to the procedure, but it should be discussed not here.