Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ImportWarning: can't resolve package from __spec__ or __package__ #3061

Closed
blueyed opened this Issue Dec 25, 2017 · 13 comments

Comments

Projects
None yet
6 participants
@blueyed
Copy link
Contributor

blueyed commented Dec 25, 2017

Given the following directory structuce:

.
└── tests
    ├── conftest.py
    ├── __init__.py
    ├── relative.py
    └── test_foo.py

With tests/conftest.py:

from .relative import relative

print("conftest")

tests/relative.py:

relative = 1

tests/test_foo.py:

def test_foo():
    pass

Running PYTHONWARNINGS=all pytest tests -s shows:

…/pyenv/tmp-system-ImportWarning-ZniFiE/lib/python3.6/site.py:165: DeprecationWarning: 'U' mode is deprecated
  f = open(fullname, "rU")
…/Vcs/pytest/_pytest/assertion/rewrite.py:7: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
…/Vcs/pytest/ImportWarning/tests/conftest.py:1: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  from .relative import relative
conftest
==================================== test session starts ====================================
platform linux -- Python 3.6.3, pytest-3.3.2.dev14+gd3b54d03.d20171225, py-1.5.2, pluggy-0.6.0
rootdir: …/Vcs/pytest, inifile: tox.ini
collected 1 item                                                                            

tests/test_foo.py .                                                                   [100%]

================================= 1 passed in 0.01 seconds ==================================

The ImportWarning is:

tests/conftest.py:1: ImportWarning: can't resolve package from spec or package, falling back on name and path
from .relative import relative

It only shows up when using -s.

This came up for pytest-django.
See pytest-dev/pytest-django#546 (comment).

Defining __package__ explicitly fixes it, but it looks like this should not be necessary.

@pytestbot

This comment has been minimized.

Copy link

pytestbot commented Dec 25, 2017

GitMate.io thinks the contributor most likely able to help you is @nicoddemus.

@asottile

This comment has been minimized.

Copy link
Member

asottile commented Dec 29, 2017

Now that pytest-xdist uses explicit relative imports (after pytest-dev/pytest-xdist#242) this is also breaking the py36-xdist suite for pytest's tests.

nicoddemus added a commit to nicoddemus/pytest that referenced this issue Jan 8, 2018

Ignore ImportWarnings regarding package resolution
The problem is described/discussed in pytest-dev#3061

Ideally this should be a temporary solution until we find a proper one
which gets rid of the warning
@nicoddemus

This comment has been minimized.

Copy link
Member

nicoddemus commented Jan 8, 2018

Opened #3096 to at least ignore the warning for now so we can get clean builds again.

Anybody knows what's the proper fix here? Who's doing something wrong, pytest or pytest-xdist?

@asottile

This comment has been minimized.

Copy link
Member

asottile commented Jan 8, 2018

without any evidence my guess would be pytest (and if I were to extend that guess further, it would go away with assert rewriting off).

Let's validate those guesses:

$ PYTHONWARNINGS=all py.test tests -s --assert=plain >& assert_plain.txt
$ PYTHONWARNINGS=all py.test tests -s >& assert_rewrite.txt
$ diff -u assert_*.txt
--- assert_plain.txt	2018-01-08 14:42:20.156677858 -0800
+++ assert_rewrite.txt	2018-01-08 14:42:28.528677858 -0800
@@ -24,3 +24,5 @@
   params = attr.ib(convert=attr.converters.optional(tuple))
 /tmp/foo/venv/lib/python3.6/site-packages/_pytest/fixtures.py:846: DeprecationWarning: The `convert` argument is deprecated in favor of `converter`.  It will be removed after 2019/01.
   ids = attr.ib(default=None, convert=_ensure_immutable_ids)
+/tmp/foo/tests/conftest.py:1: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
+  from .relative import relative

🤔 hmmm... indeed.

Then it would probably be the way pytest is building module objects for __init__ files (with incorrect __package__)?

@asottile

This comment has been minimized.

Copy link
Member

asottile commented Jan 8, 2018

probably worth reading as well: https://www.python.org/dev/peps/pep-0451/

nicoddemus added a commit to nicoddemus/pytest that referenced this issue Jan 8, 2018

Ignore ImportWarnings regarding package resolution
The problem is described/discussed in pytest-dev#3061

Ideally this should be a temporary solution until we find a proper one
which gets rid of the warning
@BoboTiG

This comment has been minimized.

Copy link

BoboTiG commented Jun 15, 2018

I have similar warnings, let me know if it needs a separate issue or a pytest-cov issue.

platform win32 -- Python 3.6.5, pytest-3.6.1, py-1.5.3, pluggy-0.6.0 -- C:\python-3.6.5\python.exe
plugins: cov-2.5.1

C:\python-3.6.5\lib\site-packages\pytest_cov\plugin.py:8: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  from . import embed

C:\python-3.6.5\lib\site-packages\pytest_cov\plugin.py:9: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  from . import engine

C:\python-3.6.5\lib\site-packages\pytest_cov\engine.py:11: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  from .compat import StringIO

C:\python-3.6.5\lib\site-packages\pytest_cov\plugin.py:10: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  from . import compat
@RonnyPfannschmidt

This comment has been minimized.

Copy link
Member

RonnyPfannschmidt commented Jun 15, 2018

its the same problem -

@blueyed

This comment has been minimized.

Copy link
Contributor Author

blueyed commented Jun 23, 2018

Ping @flub - any idea?

@asottile

This comment has been minimized.

Copy link
Member

asottile commented Jun 23, 2018

This fixes the warning for me in python3.6:

diff --git a/src/_pytest/assertion/rewrite.py b/src/_pytest/assertion/rewrite.py
index eceed611..bc1f058e 100644
--- a/src/_pytest/assertion/rewrite.py
+++ b/src/_pytest/assertion/rewrite.py
@@ -4,6 +4,7 @@ import ast
 import errno
 import itertools
 import imp
+import importlib
 import marshal
 import os
 import re
@@ -207,7 +208,13 @@ class AssertionRewritingHook(object):
         # I wish I could just call imp.load_compiled here, but __file__ has to
         # be set properly. In Python 3.2+, this all would be handled correctly
         # by load_compiled.
-        mod = sys.modules[name] = imp.new_module(name)
+        if sys.version_info >= (3, 5):
+            spec = importlib.util.spec_from_file_location(
+                name, co.co_filename, loader=self,
+            )
+            mod = sys.modules[name] = importlib.util.module_from_spec(spec)
+        else:
+            mod = sys.modules[name] = imp.new_module(name)
         try:
             mod.__file__ = co.co_filename
             # Normally, this attribute is 3.2+.

The (unfortunate) bit is importlib.util.module_from_spec is py35+

I was able to reproduce by installing pytest-cov and using python -Wonce -mpytest test.py (with a trivial test file).

Another way was editing pytest.py in site-packages to have this code at the top:

import warnings
warnings.filter('error', ImportWarning)

which produces this trace:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/t/venv/lib/python3.6/site-packages/pytest.py", line 69, in <module>
    raise SystemExit(pytest.main())
  File "/tmp/t/venv/lib/python3.6/site-packages/_pytest/config/__init__.py", line 56, in main
    config = _prepareconfig(args, plugins)
  File "/tmp/t/venv/lib/python3.6/site-packages/_pytest/config/__init__.py", line 181, in _prepareconfig
    pluginmanager=pluginmanager, args=args
  File "/tmp/t/venv/lib/python3.6/site-packages/pluggy/__init__.py", line 617, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/tmp/t/venv/lib/python3.6/site-packages/pluggy/__init__.py", line 222, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/tmp/t/venv/lib/python3.6/site-packages/pluggy/__init__.py", line 216, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "/tmp/t/venv/lib/python3.6/site-packages/pluggy/callers.py", line 196, in _multicall
    gen.send(outcome)
  File "/tmp/t/venv/lib/python3.6/site-packages/_pytest/helpconfig.py", line 89, in pytest_cmdline_parse
    config = outcome.get_result()
  File "/tmp/t/venv/lib/python3.6/site-packages/pluggy/callers.py", line 76, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/tmp/t/venv/lib/python3.6/site-packages/pluggy/callers.py", line 180, in _multicall
    res = hook_impl.function(*args)
  File "/tmp/t/venv/lib/python3.6/site-packages/_pytest/config/__init__.py", line 607, in pytest_cmdline_parse
    self.parse(args)
  File "/tmp/t/venv/lib/python3.6/site-packages/_pytest/config/__init__.py", line 772, in parse
    self._preparse(args, addopts=addopts)
  File "/tmp/t/venv/lib/python3.6/site-packages/_pytest/config/__init__.py", line 724, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/tmp/t/venv/lib/python3.6/site-packages/pluggy/__init__.py", line 397, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/tmp/t/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2318, in load
    return self.resolve()
  File "/tmp/t/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2324, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/tmp/t/venv/lib/python3.6/site-packages/_pytest/assertion/rewrite.py", line 217, in load_module
    py.builtin.exec_(co, mod.__dict__)
  File "/tmp/t/venv/lib/python3.6/site-packages/pytest_cov/plugin.py", line 8, in <module>
    from . import embed
ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
@asottile

This comment has been minimized.

Copy link
Member

asottile commented Jun 23, 2018

if I were to turn that into a patch I'd probably (at the module scope) do something like:

if sys,version_info >= (3, 5):
    import importlib.util
    def new_module(name, path, loader):
        ...
else:
    def new_module(name, path, loader):
        ...

since I should really be importing importlib.util and not depending on import side-effects by just importing importlib (and importlib.util doesn't exist in all the versions of python pytest supports, notably 2.7)

lmk and I can craft this into a patch -- no idea how to regression test it though 🙃

@asottile

This comment has been minimized.

Copy link
Member

asottile commented Jun 23, 2018

alternatively, I wonder if assigning __spec__ before calling exec_ would also resolve this warning 🤔

@asottile

This comment has been minimized.

Copy link
Member

asottile commented Jun 23, 2018

indeed, this also silences the warning:

@@ -213,6 +214,10 @@ class AssertionRewritingHook(object):
             # Normally, this attribute is 3.2+.
             mod.__cached__ = pyc
             mod.__loader__ = self
+            if sys.version_info >= (3, 5):
+                mod.__spec__ = importlib.util.spec_from_file_location(
+                    name, co.co_filename, loader=self,
+                )
             py.builtin.exec_(co, mod.__dict__)
         except:  # noqa
             if name in sys.modules:
@blueyed

This comment has been minimized.

Copy link
Contributor Author

blueyed commented Jun 23, 2018

Awesome, thanks!

The following at the top of the module makes sense then IMHO:

if sys.version_info >= (3, 5):
    from importlib.util import spec_from_file_location

jezdez added a commit to mozilla/telemetry-analysis-service that referenced this issue Aug 14, 2018

Update pytest to 3.7.1 (#1247)
This PR updates [pytest](https://pypi.org/project/pytest) from **3.6.2** to **3.7.1**.



<details>
  <summary>Changelog</summary>
  
  
   ### 3.7.1
   ```
   =========================

Bug Fixes
---------

- `3473 &lt;https://github.com/pytest-dev/pytest/issues/3473&gt;`_: Raise immediately if ``approx()`` is given an expected value of a type it doesn&#39;t understand (e.g. strings, nested dicts, etc.).


- `3712 &lt;https://github.com/pytest-dev/pytest/issues/3712&gt;`_: Correctly represent the dimensions of an numpy array when calling ``repr()`` on ``approx()``.

- `3742 &lt;https://github.com/pytest-dev/pytest/issues/3742&gt;`_: Fix incompatibility with third party plugins during collection, which produced the error ``object has no attribute &#39;_collectfile&#39;``.

- `3745 &lt;https://github.com/pytest-dev/pytest/issues/3745&gt;`_: Display the absolute path if ``cache_dir`` is not relative to the ``rootdir`` instead of failing.


- `3747 &lt;https://github.com/pytest-dev/pytest/issues/3747&gt;`_: Fix compatibility problem with plugins and the warning code issued by fixture functions when they are called directly.


- `3748 &lt;https://github.com/pytest-dev/pytest/issues/3748&gt;`_: Fix infinite recursion in ``pytest.approx`` with arrays in ``numpy&lt;1.13``.


- `3757 &lt;https://github.com/pytest-dev/pytest/issues/3757&gt;`_: Pin pathlib2 to ``&gt;=2.2.0`` as we require ``__fspath__`` support.


- `3763 &lt;https://github.com/pytest-dev/pytest/issues/3763&gt;`_: Fix ``TypeError`` when the assertion message is ``bytes`` in python 3.
   ```
   
  
  
   ### 3.7.0
   ```
   =========================

Deprecations and Removals
-------------------------

- `2639 &lt;https://github.com/pytest-dev/pytest/issues/2639&gt;`_: ``pytest_namespace`` has been deprecated.

  See the documentation for ``pytest_namespace`` hook for suggestions on how to deal
  with this in plugins which use this functionality.


- `3661 &lt;https://github.com/pytest-dev/pytest/issues/3661&gt;`_: Calling a fixture function directly, as opposed to request them in a test function, now issues a ``RemovedInPytest4Warning``. It will be changed into an error in pytest ``4.0``.

  This is a great source of confusion to new users, which will often call the fixture functions and request them from test functions interchangeably, which breaks the fixture resolution model.



Features
--------

- `2283 &lt;https://github.com/pytest-dev/pytest/issues/2283&gt;`_: New ``package`` fixture scope: fixtures are finalized when the last test of a *package* finishes. This feature is considered **experimental**, so use it sparingly.


- `3576 &lt;https://github.com/pytest-dev/pytest/issues/3576&gt;`_: ``Node.add_marker`` now supports an ``append=True/False`` parameter to determine whether the mark comes last (default) or first.


- `3579 &lt;https://github.com/pytest-dev/pytest/issues/3579&gt;`_: Fixture ``caplog`` now has a ``messages`` property, providing convenient access to the format-interpolated log messages without the extra data provided by the formatter/handler.


- `3610 &lt;https://github.com/pytest-dev/pytest/issues/3610&gt;`_: New ``--trace`` option to enter the debugger at the start of a test.


- `3623 &lt;https://github.com/pytest-dev/pytest/issues/3623&gt;`_: Introduce ``pytester.copy_example`` as helper to do acceptance tests against examples from the project.



Bug Fixes
---------

- `2220 &lt;https://github.com/pytest-dev/pytest/issues/2220&gt;`_: Fix a bug where fixtures overridden by direct parameters (for example parametrization) were being instantiated even if they were not being used by a test.


- `3695 &lt;https://github.com/pytest-dev/pytest/issues/3695&gt;`_: Fix ``ApproxNumpy`` initialisation argument mixup, ``abs`` and ``rel`` tolerances were flipped causing strange comparsion results.
  Add tests to check ``abs`` and ``rel`` tolerances for ``np.array`` and test for expecting ``nan`` with ``np.array()``


- `980 &lt;https://github.com/pytest-dev/pytest/issues/980&gt;`_: Fix truncated locals output in verbose mode.



Improved Documentation
----------------------

- `3295 &lt;https://github.com/pytest-dev/pytest/issues/3295&gt;`_: Correct the usage documentation of ``--last-failed-no-failures`` by adding the missing ``--last-failed`` argument in the presented examples, because they are misleading and lead to think that the missing argument is not needed.



Trivial/Internal Changes
------------------------

- `3519 &lt;https://github.com/pytest-dev/pytest/issues/3519&gt;`_: Now a ``README.md`` file is created in ``.pytest_cache`` to make it clear why the directory exists.
   ```
   
  
  
   ### 3.6.4
   ```
   =========================

Bug Fixes
---------

- Invoke pytest using ``-mpytest`` so ``sys.path`` does not get polluted by packages installed in ``site-packages``. (`742 &lt;https://github.com/pytest-dev/pytest/issues/742&gt;`_)


Improved Documentation
----------------------

- Use ``smtp_connection`` instead of ``smtp`` in fixtures documentation to avoid possible confusion. (`3592 &lt;https://github.com/pytest-dev/pytest/issues/3592&gt;`_)


Trivial/Internal Changes
------------------------

- Remove obsolete ``__future__`` imports. (`2319 &lt;https://github.com/pytest-dev/pytest/issues/2319&gt;`_)

- Add CITATION to provide information on how to formally cite pytest. (`3402 &lt;https://github.com/pytest-dev/pytest/issues/3402&gt;`_)

- Replace broken type annotations with type comments. (`3635 &lt;https://github.com/pytest-dev/pytest/issues/3635&gt;`_)

- Pin ``pluggy`` to ``&lt;0.8``. (`3727 &lt;https://github.com/pytest-dev/pytest/issues/3727&gt;`_)
   ```
   
  
  
   ### 3.6.3
   ```
   =========================

Bug Fixes
---------

- Fix ``ImportWarning`` triggered by explicit relative imports in
  assertion-rewritten package modules. (`3061
  &lt;https://github.com/pytest-dev/pytest/issues/3061&gt;`_)

- Fix error in ``pytest.approx`` when dealing with 0-dimension numpy
  arrays. (`3593 &lt;https://github.com/pytest-dev/pytest/issues/3593&gt;`_)

- No longer raise ``ValueError`` when using the ``get_marker`` API. (`3605
  &lt;https://github.com/pytest-dev/pytest/issues/3605&gt;`_)

- Fix problem where log messages with non-ascii characters would not
  appear in the output log file.
  (`3630 &lt;https://github.com/pytest-dev/pytest/issues/3630&gt;`_)

- No longer raise ``AttributeError`` when legacy marks can&#39;t be stored in
  functions. (`3631 &lt;https://github.com/pytest-dev/pytest/issues/3631&gt;`_)


Improved Documentation
----------------------

- The description above the example for ``pytest.mark.skipif`` now better
  matches the code. (`3611
  &lt;https://github.com/pytest-dev/pytest/issues/3611&gt;`_)


Trivial/Internal Changes
------------------------

- Internal refactoring: removed unused ``CallSpec2tox ._globalid_args``
  attribute and ``metafunc`` parameter from ``CallSpec2.copy()``. (`3598
  &lt;https://github.com/pytest-dev/pytest/issues/3598&gt;`_)

- Silence usage of ``reduce`` warning in Python 2 (`3609
  &lt;https://github.com/pytest-dev/pytest/issues/3609&gt;`_)

- Fix usage of ``attr.ib`` deprecated ``convert`` parameter. (`3653
  &lt;https://github.com/pytest-dev/pytest/issues/3653&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pytest
  - Changelog: https://pyup.io/changelogs/pytest/
  - Repo: https://github.com/pytest-dev/pytest/issues
  - Homepage: http://pytest.org
</details>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.