'collections.abc' is no longer defined when collections is imported

[bitdancer]

BPO	20784
Nosy	@warsaw, @rhettinger, @ncoghlan, @pitrou, @vstinner, @larryhastings, @tiran, @bitdancer, @ericsnowcurrently, @serhiy-storchaka, @pawciobiel

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-09-08.22:49:26.872>
created_at = <Date 2014-02-26.18:25:38.898>
labels = ['type-bug', 'library']
title = "'collections.abc' is no longer defined when collections is imported"
updated_at = <Date 2018-04-11.11:05:20.144>
user = 'https://github.com/bitdancer'

bugs.python.org fields:

activity = <Date 2018-04-11.11:05:20.144>
actor = 'pawciobiel'
assignee = 'none'
closed = True
closed_date = <Date 2016-09-08.22:49:26.872>
closer = 'christian.heimes'
components = ['Library (Lib)']
creation = <Date 2014-02-26.18:25:38.898>
creator = 'r.david.murray'
dependencies = []
files = []
hgrepos = []
issue_num = 20784
keywords = ['3.4regression']
message_count = 18.0
messages = ['212284', '212286', '212288', '212289', '212291', '212293', '212294', '212302', '212305', '213126', '213128', '213129', '213134', '213135', '213136', '213411', '213412', '315190']
nosy_count = 14.0
nosy_names = ['barry', 'rhettinger', 'ncoghlan', 'pitrou', 'vstinner', 'larry', 'christian.heimes', 'Arfrever', 'r.david.murray', 'Jurko.Gospodneti\xc4\x87', 'python-dev', 'eric.snow', 'serhiy.storchaka', 'pawciobiel']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue20784'
versions = ['Python 2.7', 'Python 3.4']

[bitdancer]

collections.abc was renamed _collections_abc in bpo-19218. The __init__ file was modified to load all the abc into the collections namespace, but the 'abc' name itself is no longer defined:

Python 3.3.2 (default, Dec 17 2013, 17:24:42) 
[GCC 4.7.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import collections
>>> collections.abc
<module 'collections.abc' from '/usr/lib/python3.3/collections/abc.py'>

Python 3.4.0rc1+ (default:1bc585ba5df2, Feb 24 2014, 15:04:31) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import collections
>>> collections.abc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'abc'

It looks like the import statement for _collections_abc in the __init__ file as has missing "as abc" phrase.

This is probably not important enough to require fixing in the RC, since 'import collections.abc' works fine, but it is a regression.

[larryhastings]

I'm pretty sure that if you import "x", there are zero guarantees that "x.y" will work. The offical line is that you must explicitly import all the deepest submodules you use. So I don't think this is even a bug.

[bitdancer]

It is a backward compatibility bug. Something that used to work doesn't any more. And it was explicitly *made* to work previously (the original __init__ statement was 'import collections.abc'). And it is is an implementation bug in the original patch because otherwise there would be no point in importing _collections_abc in __init__.

But you are right, it is relatively unlikely that anyone is relying on it.

[bitdancer]

If we decide we want this (small) backward compatibility break, to make collections.abc consistent with the other modules (except os.path), then I should mention it in the whatsnew porting section for 3.4, which is really why I opened this issue :)

I that case, IMO, the import for _collections_abc should be removed from the collections __init__ file (in 3.4.1), just to keep things tidy.

[pitrou]

I'm pretty sure that if you import "x", there are zero guarantees that
"x.y" will work. The offical line is that you must explicitly import
all the deepest submodules you use.

I'm not sure why you're saying that. I think it's quite common to only
"import os" and then use os.path.

[pitrou]

A quick grep indicates at least the following modules use os.path but only import os:
bdb, binhex, cgitb, compileall, cProfile, doctest, filecmp, fileinput, fnmatch, ftplib, gettext, glob, imghdr, imp, inspect, linecache, mailbox, mimetypes, modulefinder, netrc, optparse, pdb, platform, profile, pstats, pyclbr, pydoc, shlex, site, sndhdr, ssl, subprocess, tabnanny, tarfile, trace, uuid, uu, webbrowser, zipfile.

So, it's a very common idiom.

[serhiy-storchaka]

May be add temporary replacement for collections.abc?

class _AbcModulePlaceholder(type(_collections_abc)):
    def __warn(self):
        import warnings
        warnings.warn('collections.abc used without importing',
                      DeprecationWarning, 3)
    def __getattr__(self, name):
        self.__warn()
        return getattr(_collections_abc, name)
    def __setattr__(self, name, value):
        self.__warn()
        setattr(_collections_abc, name, value)
    def __delattr__(self, name):
        self.__warn()
        delattr(_collections_abc, name)
    def __dir__(self):
        self.__warn()
        return dir(_collections_abc)

abc = _AbcModulePlaceholder('abc')

[larryhastings]

So, it's a very common idiom.

"Common" doesn't imply "correct" or "supported". There are plenty of other packages/modules who don't import their subpackages/submodules during initialization. Unless explicitly supported by the module, using a submodule without explicitly importing it is relying on undefined behavior.

I stand by my statement.

[pitrou]

> So, it's a very common idiom.

"Common" doesn't imply "correct" or "supported". There are plenty of
other packages/modules who don't import their subpackages/submodules
during initialization. Unless explicitly supported by the module,
using a submodule without explicitly importing it is relying on
undefined behavior.

Yes, you are technically right. But this is an idealized view of what
backwards compatibility means in the real world.

Well, let's see if people complain after 3.4 is released, anyway.

[JurkoGospodneti]

Item possibly related to this. When packaging a simple HelloWorld-like application like this:

  print("Hello world")
  import configparser

using cx_Freeze and Python 3.4, you get the following error on packaged application startup:

Traceback (most recent call last):
  File "C:\Program Files\Python\Python34rc3\lib\site-packages\cx_freeze-4.3.2-py3.4-win-amd64.egg\cx_Freeze\initscripts\Console3.py", line 27, in <module>
    exec(code, m.__dict__)
  File "hello_world.py", line 4, in <module>
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2203, in _find_and_load_unlocked
    module = _SpecMethods(spec)._load_unlocked()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1191, in _load_unlocked
    return self._load_backward_compatible()
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 1161, in _load_backward_compatible
    spec.loader.load_module(spec.name)
  File "C:\Program Files\Python\Python34rc3\lib\configparser.py", line 121, in <module>
    from collections.abc import MutableMapping
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "C:\Program Files\Python\Python34rc3\lib\importlib\_bootstrap.py", line 2201, in _find_and_load_unlocked
    raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named 'collections.abc'

As I see it, the config parser module is attempting to import
'collections.abc'.

The same does not occur with Python 3.3.5 or 3.3.3.

Not sure if this is due to something cx_Freeze does so it failed to collect some module in the created installation package, or if it's something to look into in Python itself. Just looked similar to this so I bring it up as additional info for anyone looking deeper into this issue.

Hope this helps.

Best regards,
Jurko Gospodnetić

[vstinner]

Item possibly related to this.

I cannot reproduce your issue, it looks like a bug in cx_Freeze. "import collections" doesn't import "collections.abc" by default anymore, but configparser is correct: "from collections.abc import MutableMapping". So it imports explicitly "collections.abc".

[JurkoGospodneti]

I cannot reproduce your issue

Meaning you do not have the environment set up for this or that you tried it and it worked for you?

If it 'worked for you', I can send you more detailed environment information when get back to my office in an hour or so.

[vstinner]

> I cannot reproduce your issue
Meaning you do not have the environment set up for this or that
you tried it and it worked for you?

I mean that executing the following lines in Python doesn't fail:
---

print("Hello world")
import configparser

---

It's specific to cx_Freeze and unrelated to this issue.

[ncoghlan]

cx-freeze feedback is better added to bpo-20884 - I currently still suspect that 3.4 has just uncovered some latent spec non-compliance in the cx-freeze import system emulation (although I need to investigate further to be sure, since it's also possible we accidentally broke backwards compatibility with certain kinds of legacy behaviour when implementing PEP-451).

For this issue, I don't think os.path can be used as a precedent - "os" isn't a package, and "os.path" isn't a normal submodule (instead, the os module does a dance to figure out which top level module to import as path - it's usually either ntpath or posixpath).

Instead, this is just normal submodule import behaviour - "import collections" doesn't necessarily imply "import collections.abc", it just used to do so because that was how the backwards compatibility for the old names happened to be implemented.

Setting the attribute would also shadow the submodule, which would be a little weird (since collections.abc could then refer to either that module or to _collections_abc). Really, the change to collections/init.py could just be reverted - the _collections_abc move was just to avoid running collections/init.py at startup, so there's no need to use the hack inside collections/init.py itself.

[ncoghlan]

Oh, and yes, this is *definitely* a bug: _collections_abc.__name__ is set to "collections.abc", but if "collections.abc" isn't imported anywhere in the program, then cx-freeze and similar tools will miss the fact that "collections.abc" should be bundled.

If "collections" is changed back to implicitly importing the submodule, that problem should go away (although the lie in __name__ for pickle compatibility may still cause problems with freezing, so it's perhaps worth mentioning in the porting notes regardless).

[python-dev]

New changeset d575398d1916 by R David Murray in branch 'default':
whatsnew: collections no longer implicitly imports 'abc' (bpo-20784).
http://hg.python.org/cpython/rev/d575398d1916

[bitdancer]

I've documented that 'collections.abc' is no longer implicit, which I presume means we are going to keep this behavior. (Unless you tell me to revert that and we fix it as a regression in 3.4.1).

As long as an application follows the note and explicitly imports collections.abc, I would think that the freeze tools would do the right thing. I'd think they'd do the right thing anyway, since _collections_abc appears in an 'import from' in collections.__init__, so I'm not clear what problem is anticipated for freeze tools that is different from the one that any program relying on the implicit import would face.

[pawciobiel]

I had similar issue and it helped when as a workaround I use
'pip install .' as opposed to 'python setup.py develop' in my project.
I hope this will give you a hint.