Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

importlib_metadata breaks TravisCI builds #205

Closed
Lawouach opened this issue May 7, 2019 · 25 comments · Fixed by #206 or #215
Closed

importlib_metadata breaks TravisCI builds #205

Lawouach opened this issue May 7, 2019 · 25 comments · Fixed by #206 or #215

Comments

@Lawouach
Copy link

Lawouach commented May 7, 2019

Hi all,

I've noticed my builds started to fail because importlib_metadata couldn't not be found by pluggy:

Here is an example https://travis-ci.org/chaostoolkit-incubator/chaostoolkit-aws/jobs/529280278

running build_ext
Traceback (most recent call last):
  File "setup.py", line 97, in <module>
    main()
  File "setup.py", line 93, in main
    setuptools.setup(**setup_params)
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/opt/python/3.6.3/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/opt/python/3.6.3/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/opt/python/3.6.3/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/travis/build/chaostoolkit-incubator/chaostoolkit-aws/.eggs/pytest_runner-4.4-py3.6.egg/ptr.py", line 190, in run
    return self.run_tests()
  File "/home/travis/build/chaostoolkit-incubator/chaostoolkit-aws/.eggs/pytest_runner-4.4-py3.6.egg/ptr.py", line 201, in run_tests
    result_code = __import__('pytest').main()
  File "/home/travis/build/chaostoolkit-incubator/chaostoolkit-aws/.eggs/pytest-4.4.1-py3.6.egg/pytest.py", line 7, in <module>
    from _pytest.assertion import register_assert_rewrite
  File "/home/travis/build/chaostoolkit-incubator/chaostoolkit-aws/.eggs/pytest-4.4.1-py3.6.egg/_pytest/assertion/__init__.py", line 12, in <module>
    from _pytest.assertion import rewrite
  File "/home/travis/build/chaostoolkit-incubator/chaostoolkit-aws/.eggs/pytest-4.4.1-py3.6.egg/_pytest/assertion/rewrite.py", line 23, in <module>
    from _pytest.assertion import util
  File "/home/travis/build/chaostoolkit-incubator/chaostoolkit-aws/.eggs/pytest-4.4.1-py3.6.egg/_pytest/assertion/util.py", line 10, in <module>
    import _pytest._code
  File "/home/travis/build/chaostoolkit-incubator/chaostoolkit-aws/.eggs/pytest-4.4.1-py3.6.egg/_pytest/_code/__init__.py", line 6, in <module>
    from .code import Code  # noqa
  File "/home/travis/build/chaostoolkit-incubator/chaostoolkit-aws/.eggs/pytest-4.4.1-py3.6.egg/_pytest/_code/code.py", line 15, in <module>
    import pluggy
  File "/home/travis/build/chaostoolkit-incubator/chaostoolkit-aws/.eggs/pluggy-0.10.0-py3.6.egg/pluggy/__init__.py", line 16, in <module>
    from .manager import PluginManager, PluginValidationError
  File "/home/travis/build/chaostoolkit-incubator/chaostoolkit-aws/.eggs/pluggy-0.10.0-py3.6.egg/pluggy/manager.py", line 6, in <module>
    import importlib_metadata
  File "/home/travis/build/chaostoolkit-incubator/chaostoolkit-aws/.eggs/importlib_metadata-0.9-py3.6.egg/importlib_metadata/__init__.py", line 20, in <module>
    __version__ = version(__name__)
  File "/home/travis/build/chaostoolkit-incubator/chaostoolkit-aws/.eggs/importlib_metadata-0.9-py3.6.egg/importlib_metadata/api.py", line 348, in version
    return distribution(package).version
  File "/home/travis/build/chaostoolkit-incubator/chaostoolkit-aws/.eggs/importlib_metadata-0.9-py3.6.egg/importlib_metadata/api.py", line 313, in distribution
    return Distribution.from_name(package)
  File "/home/travis/build/chaostoolkit-incubator/chaostoolkit-aws/.eggs/importlib_metadata-0.9-py3.6.egg/importlib_metadata/api.py", line 160, in from_name
    raise PackageNotFoundError(name)
importlib_metadata.api.PackageNotFoundError: importlib_metadata

It happens on Python 3.6 and 3.7, but not on 3.5 because, on that one, pytest (which imports pluggy) remains on an older version.

But then again, I'm confused because, importlib_metadata is indeed installed first:

Searching for pluggy>=0.9
Reading https://pypi.python.org/simple/pluggy/
Downloading https://files.pythonhosted.org/packages/57/ba/54c93ac55084bde8f9195ed5cd04223b3c0d30d300801bf556565cb7675a/pluggy-0.10.0-py2.py3-none-any.whl#sha256=1c0b297d4d41bc9bdfbdc17991b35f9e1d2cfe8eaa4d7c118e86d705870d34c8
Best match: pluggy 0.10.0
Processing pluggy-0.10.0-py2.py3-none-any.whl
Installing pluggy-0.10.0-py2.py3-none-any.whl to /home/travis/build/chaostoolkit-incubator/chaostoolkit-aws/.eggs
writing requirements to /home/travis/build/chaostoolkit-incubator/chaostoolkit-aws/.eggs/pluggy-0.10.0-py3.6.egg/EGG-INFO/requires.txt
Installed /home/travis/build/chaostoolkit-incubator/chaostoolkit-aws/.eggs/pluggy-0.10.0-py3.6.egg

Bizarrely, I fail to reproduce locally.

I'm wondering if this is something you folks have run into by any chance? Likely on my side but I fail to figure out what's different.

Thanks for any tips!

@nr7s
Copy link

nr7s commented May 7, 2019

I can't also reproduce it locally but I'm getting the exact same error. I'm fixing the version to 0.9.0 for now on my setup.py.

Luckily I realized this quick as my CI was working fine and on a second run 2 minutes later or something it stopped working because of this.

@Lawouach
Copy link
Author

Lawouach commented May 7, 2019

At least, I'm not going nuts :p

But this is still a mystery to me.

@RonnyPfannschmidt
Copy link
Member

i see that you have a lot of eggs installed, its not clear to me whether importlib-metadata does actually support eggs

@Lawouach
Copy link
Author

Lawouach commented May 7, 2019

Their README states they don't indeed. I guess, in that case, I should likely open an issue there, right?

I'll pin pluggy to 0.9.0 for now unfortunately.

@Lawouach
Copy link
Author

Lawouach commented May 7, 2019

@RonnyPfannschmidt
Copy link
Member

@asottile i bevel this makes importlib-metadata a absolutely breaking change until that's sorted out

Lawouach added a commit to chaostoolkit-incubator/chaostoolkit-aws that referenced this issue May 7, 2019
See pytest-dev/pluggy#205

Signed-off-by: Sylvain Hellegouarch <sh@defuze.org>
@nicoddemus
Copy link
Member

@asottile i bevel this makes importlib-metadata a absolutely breaking change until that's sorted out

Hmmm indeed, this will be a problem. Let's see what @asottile has to say, otherwise I believe we need to revert #199 and make a 0.10.1 release.

@RonnyPfannschmidt
Copy link
Member

i would like to see the metadata backed extracted into something configurable
while that is being done, there can also be a abstraction for back-end distributions that exposes the interesting details in a encapsulated manner,

@Lawouach
Copy link
Author

Lawouach commented May 7, 2019

So, while investigating how to get a correct set of test package depedencies to work, I'm now back to older versions of pluggy that failed with this one pytest-dev/pytest#3753

I don't doubt, I'll get the right combo at some point.

@alex
Copy link

alex commented May 7, 2019

We're seeing this on pyca/cryptography, only in our setup.py test CI builds on Jenkins. Builds that directly invoke pytest inside of tox do not seem to be impacted.

@asottile
Copy link
Member

asottile commented May 7, 2019

yeah seems right, a shame that we still have to support eggs in 2019

@Lawouach
Copy link
Author

Lawouach commented May 7, 2019

Agreed. To be fair, I didn't even realise I was using them so i'm a little confused. I need to investigate.

@asottile
Copy link
Member

asottile commented May 7, 2019

I'm of the opinion we shouldn't try and support eggs, but I understand that's too much of a break for most people so we should revert this and we'll continue in the slow path indefinitely

@alex
Copy link

alex commented May 7, 2019

FWIW, on the pyca/cryptography front the reason we support setup.py test is because it represents a common entrypoint that Linux distros who redistribute us like.

If setup.py test is truly the last reason to care about .egg files, maybe we need to work with Linux distros to stop using that so we can kill setup.py test.

@asottile
Copy link
Member

asottile commented May 7, 2019

there's other ways to use pytest via setup.py test without using eggs, they're just a little more work (setting up a virtualenv, invoking in that, etc.) and sadly not the default

@alex
Copy link

alex commented May 7, 2019

https://github.com/pyca/cryptography/blob/master/setup.py#L71-L90 is our current setup.py test integration. How much extra code is it to make a virtualenv happen :-)

@Lawouach
Copy link
Author

Lawouach commented May 7, 2019

What is confusing to me is that I have many other projects, copycats of this one and they succeeded just fine so why did this one decided to fail today? It actually worked fine two weeks ago...

@alex
Copy link

alex commented May 7, 2019

🤷‍♂

@asottile
Copy link
Member

asottile commented May 7, 2019

It only started failing today as we released pluggy 0.10.0 which switched the internal mechanism from pkg_resources to importlib-metadata

@Lawouach
Copy link
Author

Lawouach commented May 7, 2019

Ha! I didn't realise you'd released today.

@asottile
Copy link
Member

asottile commented May 7, 2019

0.11.0 has been released -- sorry again for the breakage!

I'll be following up with documenting how to avoid eggs in setup.py test for pytest and perhaps even implementing egg support in importlib-metadata.

@nicoddemus
Copy link
Member

Great, thanks @asottile for the lightning fast response!

@Lawouach
Copy link
Author

Lawouach commented May 7, 2019

Indeed. Thank you @asottile!

@nr7s
Copy link

nr7s commented May 7, 2019

Thanks @asottile !

NyanKiyoshi pushed a commit to NyanKiyoshi/pytest-django-queries that referenced this issue May 7, 2019
sitaktif added a commit to sitaktif/pytest that referenced this issue May 8, 2019
Pluggy 0.10.0 introduced a new dependency `importlib-metadata` but that package cannot be imported when installed as an egg and that is causing problems for users relying on `setup.py` to install packages present in `test_require`. This change was quickly8 reverted in pluggy 0.11.0.

See pytest-dev/pluggy#205
sitaktif added a commit to sitaktif/pytest that referenced this issue May 8, 2019
Pluggy 0.10.0 introduced a new dependency `importlib-metadata` but that
package cannot be imported when installed as an egg and that is causing
problems for users relying on `setup.py` to install packages present in
`test_require`. This change was quickly reverted in pluggy 0.11.0.

See pytest-dev/pluggy#205
bors bot added a commit to mozilla/normandy that referenced this issue May 17, 2019
1887: Scheduled weekly dependency update for week 19 r=rehandalal a=pyup-bot






### Update [botocore](https://pypi.org/project/botocore) from **1.12.142** to **1.12.146**.


<details>
  <summary>Changelog</summary>
  
  
   ### 1.12.146
   ```
   ========

* api-change:``glue``: Update glue client to latest version
* api-change:``sts``: Update sts client to latest version
   ```
   
  
  
   ### 1.12.145
   ```
   ========

* api-change:``servicecatalog``: Update servicecatalog client to latest version
* api-change:``eks``: Update eks client to latest version
* api-change:``sagemaker``: Update sagemaker client to latest version
* api-change:``kinesisanalytics``: Update kinesisanalytics client to latest version
* api-change:``kinesisanalyticsv2``: Update kinesisanalyticsv2 client to latest version
   ```
   
  
  
   ### 1.12.144
   ```
   ========

* api-change:``appsync``: Update appsync client to latest version
* api-change:``storagegateway``: Update storagegateway client to latest version
* api-change:``ssm``: Update ssm client to latest version
* api-change:``alexaforbusiness``: Update alexaforbusiness client to latest version
   ```
   
  
  
   ### 1.12.143
   ```
   ========

* api-change:``config``: Update config client to latest version
* api-change:``iam``: Update iam client to latest version
* api-change:``sts``: Update sts client to latest version
* api-change:``codepipeline``: Update codepipeline client to latest version
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/botocore
  - Changelog: https://pyup.io/changelogs/botocore/
  - Repo: https://github.com/boto/botocore
</details>





### Update [Faker](https://pypi.org/project/Faker) from **1.0.5** to **1.0.6**.


*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/faker
  - Changelog: https://pyup.io/changelogs/faker/
  - Repo: https://github.com/joke2k/faker
</details>





### Update [pluggy](https://pypi.org/project/pluggy) from **0.9.0** to **0.11.0**.


<details>
  <summary>Changelog</summary>
  
  
   ### 0.11.0
   ```
   ==========================

Bug Fixes
---------

- `205 &lt;https://github.com/pytest-dev/pluggy/issues/205&gt;`_: Revert changes made in 0.10.0 release breaking ``.egg`` installs.
   ```
   
  
  
   ### 0.10.0
   ```
   ==========================

Features
--------

- `199 &lt;https://github.com/pytest-dev/pluggy/issues/199&gt;`_: Switch from ``pkg_resources`` to ``importlib-metadata`` for entrypoint detection for improved performance and import time.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pluggy
  - Changelog: https://pyup.io/changelogs/pluggy/
  - Repo: https://github.com/pytest-dev/pluggy
</details>





### Update [Pygments](https://pypi.org/project/Pygments) from **2.3.1** to **2.4.0**.


*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pygments
  - Homepage: http://pygments.org/
</details>





### Update [pyrsistent](https://pypi.org/project/pyrsistent) from **0.15.1** to **0.15.2**.


<details>
  <summary>Changelog</summary>
  
  
   ### 0.15.2
   ```
   * Fix 166, Propagate &#39;ignore_extra&#39; param in hierarchy. Thanks ss18 for this!
 * Fix 167, thaw typing. Thanks nattofriends for this!
 * Fix 154, not possible to insert empty pmap as leaf node with transform.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pyrsistent
  - Changelog: https://pyup.io/changelogs/pyrsistent/
  - Repo: http://github.com/tobgu/pyrsistent/
</details>





### Update [boto3](https://pypi.org/project/boto3) from **1.9.142** to **1.9.146**.


<details>
  <summary>Changelog</summary>
  
  
   ### 1.9.146
   ```
   =======

* api-change:``glue``: [``botocore``] Update glue client to latest version
* api-change:``sts``: [``botocore``] Update sts client to latest version
   ```
   
  
  
   ### 1.9.145
   ```
   =======

* api-change:``servicecatalog``: [``botocore``] Update servicecatalog client to latest version
* api-change:``eks``: [``botocore``] Update eks client to latest version
* api-change:``sagemaker``: [``botocore``] Update sagemaker client to latest version
* api-change:``kinesisanalytics``: [``botocore``] Update kinesisanalytics client to latest version
* api-change:``kinesisanalyticsv2``: [``botocore``] Update kinesisanalyticsv2 client to latest version
   ```
   
  
  
   ### 1.9.144
   ```
   =======

* api-change:``appsync``: [``botocore``] Update appsync client to latest version
* api-change:``storagegateway``: [``botocore``] Update storagegateway client to latest version
* api-change:``ssm``: [``botocore``] Update ssm client to latest version
* api-change:``alexaforbusiness``: [``botocore``] Update alexaforbusiness client to latest version
   ```
   
  
  
   ### 1.9.143
   ```
   =======

* api-change:``config``: [``botocore``] Update config client to latest version
* api-change:``iam``: [``botocore``] Update iam client to latest version
* api-change:``sts``: [``botocore``] Update sts client to latest version
* api-change:``codepipeline``: [``botocore``] Update codepipeline client to latest version
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/boto3
  - Changelog: https://pyup.io/changelogs/boto3/
  - Repo: https://github.com/boto/boto3
</details>





### Update [django-cors-headers](https://pypi.org/project/django-cors-headers) from **2.5.3** to **3.0.1**.


<details>
  <summary>Changelog</summary>
  
  
   ### 3.0.0
   ```
   ------------------

* ``CORS_ORIGIN_WHITELIST`` now requires URI schemes, and optionally ports.
  This is part of the CORS specification
  (`Section 3.2 &lt;https://tools.ietf.org/html/rfc6454section-3.2&gt;`_) that was
  not implemented in this library, except from with the
  ``CORS_ORIGIN_REGEX_WHITELIST`` setting. It fixes a security issue where the
  CORS middleware would allow requests between schemes, for example from
  insecure ``http://`` Origins to a secure ``https://`` site.

  You will need to update your whitelist to include schemes, for example from
  this:

  .. code-block:: python

      CORS_ORIGIN_WHITELIST = [&#39;example.com&#39;]

  ...to this:

  .. code-block:: python

      CORS_ORIGIN_WHITELIST = [&#39;https://example.com&#39;]

* Removed the ``CORS_MODEL`` setting, and associated class. It seems very few,
  or no users were using it, since there were no bug reports since its move to
  abstract in version 2.0.0 (2017-01-07). If you *are* using this
  functionality, you can continue by changing your model to not inherit from
  the abstract one, and add a signal handler for ``check_request_enabled`` that
  reads from your model. Note you&#39;ll need to handle the move to include schemes
  for Origins.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/django-cors-headers
  - Changelog: https://pyup.io/changelogs/django-cors-headers/
  - Repo: https://github.com/ottoyiu/django-cors-headers
</details>





### Update [djangorestframework](https://pypi.org/project/djangorestframework) from **3.9.3** to **3.9.4**.


*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/djangorestframework
  - Changelog: https://pyup.io/changelogs/djangorestframework/
  - Homepage: https://www.django-rest-framework.org/
</details>





### Update [factory_boy](https://pypi.org/project/factory_boy) from **2.11.1** to **2.12.0**.


<details>
  <summary>Changelog</summary>
  
  
   ### 2.12.0
   ```
   -------------------

*New:*

    - Add support for Python 3.7
    - Add support for Django 2.1
    - Add :attr:`~factory.fuzzy.FuzzyChoice.getter` to :class:`~factory.fuzzy.FuzzyChoice` that mimics
      the behavior of ``getter`` in :class:`~factory.Iterator`
    - Make the ``extra_kwargs`` parameter of :meth:`~factory.faker.Faker.generate` optional
    - Add :class:`~factory.RelatedFactoryList` class for one-to-many support, thanks `Sean Harrington &lt;https://github.com/seanharr11&gt;`_.
    - Make the `locale` argument for :class:`~factory.faker.Faker` keyword-only

*Bugfix:*

    - Allow renamed arguments to be optional, thanks to `Justin Crown &lt;https://github.com/mrname&gt;`_.
    - Fix `django_get_or_create` behavior when using multiple fields with `unique=True`, thanks to `YPCrumble &lt;https://github.com/YPCrumble&gt;`
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/factory-boy
  - Changelog: https://pyup.io/changelogs/factory-boy/
  - Repo: https://github.com/FactoryBoy/factory_boy
</details>





### Update [kinto-http](https://pypi.org/project/kinto-http) from **10.3.0** to **10.4.0**.


<details>
  <summary>Changelog</summary>
  
  
   ### 10.4.0
   ```
   ===================

- Add support for Bearer tokens in the CLI utilities.
- Use black for code formatting.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/kinto-http
  - Changelog: https://pyup.io/changelogs/kinto-http/
  - Repo: https://github.com/Kinto/kinto-http.py/
</details>





### Update [pytest](https://pypi.org/project/pytest) from **4.4.1** to **4.5.0**.


<details>
  <summary>Changelog</summary>
  
  
   ### 4.4.2
   ```
   =========================

Bug Fixes
---------

- `5089 &lt;https://github.com/pytest-dev/pytest/issues/5089&gt;`_: Fix crash caused by error in ``__repr__`` function with both ``showlocals`` and verbose output enabled.


- `5139 &lt;https://github.com/pytest-dev/pytest/issues/5139&gt;`_: Eliminate core dependency on &#39;terminal&#39; plugin.


- `5229 &lt;https://github.com/pytest-dev/pytest/issues/5229&gt;`_: Require ``pluggy&gt;=0.11.0`` which reverts a dependency to ``importlib-metadata`` added in ``0.10.0``.
  The ``importlib-metadata`` package cannot be imported when installed as an egg and causes issues when relying on ``setup.py`` to install test dependencies.



Improved Documentation
----------------------

- `5171 &lt;https://github.com/pytest-dev/pytest/issues/5171&gt;`_: Doc: ``pytest_ignore_collect``, ``pytest_collect_directory``, ``pytest_collect_file`` and ``pytest_pycollect_makemodule`` hooks&#39;s &#39;path&#39; parameter documented type is now ``py.path.local``


- `5188 &lt;https://github.com/pytest-dev/pytest/issues/5188&gt;`_: Improve help for ``--runxfail`` flag.



Trivial/Internal Changes
------------------------

- `5182 &lt;https://github.com/pytest-dev/pytest/issues/5182&gt;`_: Removed internal and unused ``_pytest.deprecated.MARK_INFO_ATTRIBUTE``.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pytest
  - Changelog: https://pyup.io/changelogs/pytest/
  - Homepage: https://docs.pytest.org/en/latest/
</details>







Co-authored-by: pyup-bot <github-bot@pyup.io>
Co-authored-by: Michael Cooper <mythmon@gmail.com>
@asottile
Copy link
Member

#215 is the new attempt at rolling this out -- I've made sure to validate python setup.py pytest usage with eggs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants