poetry and tox-poetry-dev-dependencies

[bparzella]

At the moment it seems like installing both the main poetry package and tox-poetry-dev-dependencies causes problems.

# poetry lock
Updating dependencies
Resolving dependencies... (4.2s)

Writing lock file

# pip install tox-poetry-dev-dependencies
Looking in indexes: https://...devpi.../+simple/
Collecting tox-poetry-dev-dependencies
  Downloading https://...devpi.../tox_poetry_dev_dependencies-0.0.1-py3-none-any.whl
Requirement already satisfied: tox in d:\data\.venv\lib\site-packages (from tox-poetry-dev-dependencies) (3.20.0)
Requirement already satisfied: importlib-metadata in d:\data\.venv\lib\site-packages (from tox-poetry-dev-dependencies) (1.1.3)
Collecting poetry-core (from tox-poetry-dev-dependencies)
  Using cached https://...devpi.../poetry_core-1.0.0a9-py2.py3-none-any.whl
Requirement already satisfied: filelock>=3.0.0 in d:\data\.venv\lib\site-packages (from tox->tox-poetry-dev-dependencies) (3.0.12)
Requirement already satisfied: six>=1.14.0 in d:\data\.venv\lib\site-packages (from tox->tox-poetry-dev-dependencies) (1.15.0)
Requirement already satisfied: virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0 in d:\data\.venv\lib\site-packages (from tox->tox-poetry-dev-dependencies) (20.0.31)
Requirement already satisfied: packaging>=14 in d:\data\.venv\lib\site-packages (from tox->tox-poetry-dev-dependencies) (20.4)
Requirement already satisfied: colorama>=0.4.1; platform_system == "Windows" in d:\data\.venv\lib\site-packages (from tox->tox-poetry-dev-dependencies) (0.4.3)
Requirement already satisfied: py>=1.4.17 in d:\data\.venv\lib\site-packages (from tox->tox-poetry-dev-dependencies) (1.9.0)
Requirement already satisfied: toml>=0.9.4 in d:\data\.venv\lib\site-packages (from tox->tox-poetry-dev-dependencies) (0.10.1)
Requirement already satisfied: pluggy>=0.12.0 in d:\data\.venv\lib\site-packages (from tox->tox-poetry-dev-dependencies) (0.13.1)
Requirement already satisfied: zipp>=0.5 in d:\data\.venv\lib\site-packages (from importlib-metadata->tox-poetry-dev-dependencies) (3.1.0)
Requirement already satisfied: importlib-resources>=1.0; python_version < "3.7" in d:\data\.venv\lib\site-packages (from virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0->tox->tox-poetry-dev-dependencies) (3.0.0)
Requirement already satisfied: distlib<1,>=0.3.1 in d:\data\.venv\lib\site-packages (from virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0->tox->tox-poetry-dev-dependencies) (0.3.1)
Requirement already satisfied: appdirs<2,>=1.4.3 in d:\data\.venv\lib\site-packages (from virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0->tox->tox-poetry-dev-dependencies) (1.4.4)
Requirement already satisfied: pyparsing>=2.0.2 in d:\data\.venv\lib\site-packages (from packaging>=14->tox->tox-poetry-dev-dependencies) (2.4.7)
poetry-core 1.0.0a9 has requirement importlib-metadata<2.0.0,>=1.7.0; python_version >= "2.7" and python_version < "2.8" or python_version >= "3.5" and python_version < "3.8", but you'll have importlib-metadata 1.1.3 which is incompatible.
Installing collected packages: poetry-core, tox-poetry-dev-dependencies
Successfully installed poetry-core-1.0.0a9 tox-poetry-dev-dependencies-0.0.1

# poetry lock

[ImportError]
cannot import name '_CURRENT_VENDOR'

I know that this might not be a problem of this package, but due to a dependency, I just want to raise awareness of this issue. Feel free to close this.

[sinoroc]

I'll have a closer look later, but my first impression is that the issue is not on our side here.

For some reason there is a rather old version of importlib-metadata already in the environment, while poetry-core requires a more recent version. You might want to run something like pip check, or pipdeptree (or deptree, disclaimer: another project of mine), to maybe gather some info about the projects installed in the environment: if there is a good reason why importlib-metadata 1.1.3 is installed, and if there is a reason why a newer version can not be installed.

Also you might want to give pip's new dependency resolver a try.

[sinoroc]

Let's close the issue for now. And of course we should reopen it, if it turns out there is something we can do about it here.

Thanks for the report.

[bparzella]

Somehow pip thinks poetry requires 1.1.3:

$ pip install -U importlib-metadata
Looking in indexes: https://...devpi.../+simple/
Collecting importlib-metadata
  Using cached https://...devpi.../importlib_metadata-1.7.0-py2.py3-none-any.whl
Requirement already up-to-date: zipp>=0.5 in d:\data\.venv\lib\site-packages (from importlib-metadata) (3.1.0)
poetry 1.0.10 has requirement importlib-metadata<1.2.0,>=1.1.3; python_version < "3.8", but you'll have importlib-metadata 1.7.0 which is incompatible.
Installing collected packages: importlib-metadata
  Found existing installation: importlib-metadata 1.1.3
    Uninstalling importlib-metadata-1.1.3:
      Successfully uninstalled importlib-metadata-1.1.3
Successfully installed importlib-metadata-1.7.0

Even after manual upgrade of importlib-metadata the error persists. To me it seems like poetry and poetry-core don't like to be installed side by side?

[sinoroc]

I see it now. Indeed, poetry and poetry-core seem to have conflicting version requirements for importlib-metadata. Thanks!

I don't know why it is so, and that is a bit unfortunate. I am afraid, there is nothing we can do about it in this project. But let's keep it in mind, that there is an issue here. Although I am not sure it is really an issue (these two things are probably not meant to be installed in the same environment).

Maybe we could depend on poetry directly instead of poetry-core. Which would also somehow make sense, since poetry-core hasn't had any official final release yet (only pre-releases), but I also consider our project here as still in its proof of concept phase, so fair enough I would say. I will still rethink this, and maybe we should really use poetry instead of poetry-core, I will investigate.

What I am wondering, is why you have both poetry and tox-poetry-dev-dependencies (well tox and its plugins actually) in the same environment. I would rather recommend keeping tox (and its plugins) in one environment and poetry in another. I don't use it myself, but a lot of people seem to recommend pipx nowadays to handle such things.

[sinoroc]

I looked a bit further into this. And it seems like poetry-core is meant to be installed alongside beta versions of poetry. Starting with version 1.1 Poetry itself will depend on poetry-core.

https://github.com/python-poetry/poetry/blob/1.1.0b3/pyproject.toml#L27

But until then they are not meant to be installed in the same environment.

[bparzella]

I have both tox (+plugin) and poetry installed, because the intended use is a cd pipeline that builds, tests and deploys a package to a local devpi server.

So the way to go here is to wait for poetry 1.1?

[mikenerone]

@bparzella I'm already using the plugin, as well as poetry for some build steps, by upgrading to the preview release of Poetry (currently 1.1.0b3). If you use pipx, then pipx install --pip-args="--pre" poetry (or upgrade). Whether you want to trust prerelease is up to you, but it's worked fine for me. The only issue I've run into is python-poetry/poetry#2888, but that's pretty easy to work around once you know about it (and almost certainly wouldn't be something that strikes at all on your CD server).

[sinoroc]

So the way to go here is to wait for poetry 1.1?

I don't have definitive answers. Not yet. The project (this plugin) is very young, there a lots of things that still need to be worked out. Additionally (and it is a coincidence, both lucky and unlucky), Poetry is about to be split up in 2 parts poetry-core and the poetry as we know it. Some of the core functionality of Poetry is being extracted into its own library so that it can be reused (you can read it about it on the page for poetry-core). But it is a work in progress, both poetry and core are in alpha or beta stage. Once poetry 1.1 is released (along with poetry-core 1.0) it should be smooth sailing again.

My recommendation is always (not just for our issue here, but in general) to isolate different tools from one another as much as possible (or at least as soon as a dependency version conflict arises). In this case I would recommend having Poetry in one environment and in another environment you would have Tox and its plugins (which would include this project here and poetry-core). But I get that it can be complicated to put in place properly for some workflows.

So a possible alternative (and probably our best chance here and now) as @mikenerone suggested is:
If you feel like early adopter, then you could install the latest beta release of poetry (see poetry's release history). This should be compatible with poetry-core's beta release and our plugin.

---

But...
To make things even more complicated (ha!), I am thinking about switching our plugin's code to using the full poetry directly instead of just the poetry-core library (because of #8). I haven't decided yet.

Anyway, I don't want to break people's workflows just for the fun of it. I appreciate people giving my code a try, and I am happy if it helps them get work done. Thanks for telling me about this issue. I will give it some more thought. And we should all keep an eye out for the release of poetry 1.1, we'll see if it makes things better for us or not.

[sinoroc]

@bparzella Mike just reminded me about Tox's requires setting. With that setting you can uninstall the plugin and its dependencies, meaning poetry-core, from the environment that contains your tox installation (and apparently poetry in your case). And instead add the following setting in your tox.ini:

[tox]
# ...
requires =
    tox-poetry-dev-dependencies

This way Tox itself will take care of installing the plugin in a dedicated environment (named .tox), so that the plugin (and poetry-core) should stay out of your way. And if I am not mistaken it should prevent the version incompatibility issue that you saw from happening again. Let us know if that helps.

@mikenerone Thanks for reminding me about the requires setting.

[bparzella]

Okay, I tested it today and it is working for me. This seems like a pretty good solution for this problem. As far as I understand the tox documentation, this option is intended to be used for plugins.

[bparzella]

Should this usage be mentioned in the documentation/readme?

[sinoroc]

Okay, I tested it today and it is working for me. This seems like a pretty good solution for this problem.

Great!

As far as I understand the tox documentation, this option is intended to be used for plugins. Should this usage be mentioned in the documentation/readme?

I want to say no... I want to avoid telling people how they should install things, because there are multiple ways to do it that depend on the specific workflows of each project or user. So without knowing the context it is a bit dangerous to tell people: "install it like that", as if it were the only one true way. Saying that it is a Tox plugin and it is available on PyPI should be enough.

If the issue comes up more often we could add it in a FAQ section though.

[sinoroc]

I close the issue for now, but feel free to add anything or ask for reopening if necessary. Thanks :)