Install removes dependencies of non-installed extras

[chimaerase]

• [x ] I am on the latest Poetry version.
• [x ] I have searched the issues of this repo and believe that this is not a duplicate.
• [ x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• Debian bullseye:
• poetry 1.2.0rc1:
• Link of a Gist with the contents of your pyproject.toml file: pyproject.toml

Issue

I've been testing 1.2.0rc1 and it seems poetry is removing dependencies of non-installed extras. My understanding of #4336 and #4242 is that poetry should only ever remove packages during an install if they're superseded by the locked versions that need to be installed, or if the --sync option is used. A workaround is to rewrite the extras as optional groups, which seems to solve the problem. If nothing else, perhaps some documentation updates are needed to mark extras as deprecated?

In my pyproject.toml, setuptools is a derived dependency of gunicorn (an optional package grouped under the web extra).

Running poetry install --no-root -vvv removes setuptools:

#8 2.645 + poetry lock --check
#8 3.115 poetry.lock is consistent with pyproject.toml.
#8 3.159 + poetry install --no-root -vvv
#8 3.515 Using virtualenv: /code/.venv
#8 3.555 Project environment contains an empty path in sys_path, ignoring.
#8 3.591 Installing dependencies from lock file
#8 3.635 
#8 3.635 Finding the necessary packages for the current system
#8 3.647 
#8 3.647 Package operations: 0 installs, 0 updates, 1 removal, 2 skipped
#8 3.647 
#8 3.648   • Removing setuptools (65.3.0)
#8 4.726   • Updating setuptools (63.2.0 -> 65.3.0): Skipped for the following reason: Not required
#8 4.727   • Installing gunicorn (20.1.0): Skipped for the following reason: Not required
#8 DONE 4.9s

I also tested re-writing the extras as groups instead, and did not get the same result. Making these changes fixed the problem:

#gunicorn = {optional = true, version = "*"}
#
#[tool.poetry.extras]
#web = [
#    "gunicorn",
#]
[tool.poetry.group.web]
optional=true

[tool.poetry.group.web.dependencies]
gunicorn = "*"

[dimbleby]

I don't see that #4336 ever had anything to say about extras, and #4242 is a different thing altogether (relating to poetry 1.1's handling of poetry 1.2 lockfiles).

Described behaviour seems correct to me: you've asked to install a package without extras, and poetry has made that happen.

[chimaerase]

IMO poetry's behavior with extras should parallel what's happening with --without dev flag as documented in #4336. From the PR description: "Any locked packages will be kept in the environment regardless of what is asked via the install command unless the --sync option is specified...Overall, this should be less surprising for end users makes working with dependency groups easier."

Also, IMO whether pyproject.toml is written using an optional group or using extras, the end result should be consistent. I think this ticket shows it isn't right now.

[dimbleby]

groups and extras are absolutely not interchangeable and you should not expect them to give identical results - them not being "consistent" here is not an issue.

[chimaerase]

Ok, thanks for your reply.

I submit that the distinction between when to use extras and when to use optional groups isn't clear in the documentation yet (especially to me, mostly a 1.1 user, who only had the option to use extras in that version). I think this important distinction re: whether dependencies get removed when the extra / optional group isn't selected is important to end users and should be clearly / prominently documented.

[dimbleby]

I see that #6230 has just gone in hopefully helping to clarify the distinction, I expect further MRs improving the docs would be welcome.

[chimaerase]

Thanks, that is a good start!

Unfortunately, I'm asking for documentation largely because I'm still unclear on the motivations for making optional groups/extras behave differently when not selected for install. I don't think clarifying that reasoning is something I can contribute to.

[dimbleby]

duplicate #5895

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.