-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
poetry-core: consider *not* to vendor dependencies #2346
Comments
Embedded/Bundled dependencies have long been a problem (JARs anyone?). Unfortunately, packaging and distributing these kinds of projects are still not straightforward (this includes golang and java packages too). To your specific concern, doesn't pip do the same thing? In cases like poetry and pip vendoring base dependencies solves a lot of usability issues to a large portion of the end-users. So, I am not sure if it makes sense to not vendor. In the specific context of fedora packages, I'd think poetry would be handled similar to what is done in python-pip.spec. |
Well, pip is really a special case in fedora, because - if I understand correctly - it's handled this way to solve bootstrap issues when upgrading to newer python versions, that doesn't apply to poetry. And I really want to avoid to have to de-bundle things from poetry(-core). |
The issue with bundled stuff is that if one of those dependencies fixes a security vulnerability, a LTS distro needs to fix it everywhere – potentially in old versions of poetry, long after poetry has moved on and introduced incompatible changes.
|
Gentoo packager here. I'm packaging As distro packagers, our core contention with bundled dependencies is exactly as described by @encukou: each bundled dependency increases attack surface. That's bad. While bundling dependencies may slightly ease the installation burden for end users installing When a major insecurity outweighs a minor convenience, the former takes precedence. |
This comment has been minimized.
This comment has been minimized.
@leycec no one is saying "But pip does this bad thing, so poetry should too!". The point of this issue is to discuss this matter and, if reasonable, figure out an alternative that makes a distro package maintainer's life easier. As @encukou points out, there are inherent risks of bundling requirements. But these risks need to be weighed against multiple factors, this includes usability, access vectors, exploitability etc. I disagree with @leycec that this is a "major insecurity" for a few reasons,
As for a "minor inconvenience", it should be noted that poetry is expected to work with multiple python versions as a common use case is in environments that have mulitple python versions managed via
@leycec, as for your last comment, All that said, we do also think there is room for improvement in the way we do things. This includes our installer To this end, any constructive feedback on this is always welcome. |
One of the Fedora Python packagers here, hello. Few notes: I completely understand and respect reasons why upstream projects like pip, pipetry or pipenv decide to bundle dependencies. The reasons are valid for upstreams. As distributors, we don't like that very much and we try to avoid that for reasons other have already noted. Not only security fixes, but also other kinds of fixes are backported in distributions and maintaining and patching 8 different requests bundled in different projects is a nightmare. I understand that @abn wants to work toward understanding our reasons and respecting them -- thanks for this. In Fedora, we keep bundled dependencies in pip out of necessity, not that we would want that. Considering that we build pip wheels and use those in venv and virtualenv in all the various Python versions we ship, debundling (devendoring) would become very complex and fragile. I can talk more about that, but it would get off topic. A reasonable compromise would be to make unbundling supported and trivial. Currently, we would need to sed/patch each import statement when we unbundle. That makes upgrading the packaged poetry a very unpleasant and error-prone task. Can we work together to make the vendoring of the dependencies easily revertible at our build time? Also, I'd like to cooperate here, not argue and fight. That isn't helpful. |
You're either using:
Quite a lot of the libraries poetry vendors are backports of Python's standard library, which is has quite strong backwards-compatibility guarantees. How hard would it be to use stdlib versions of those on Python versions that have them? For the others, it seems that:
Is that right? One idea is to have the tool put its _vendor directory in sys.path at startup (before other imports), so all the libraries are importable using standard names. That would remove the need for most patches. Does anyone foresee problems with this approach? |
Not with unpatched dependencies, I don't really. |
This sounds like a good solution that would satisfy both use cases. poetry without modifications would use the vendored dependencies, and distros could easily remove the sys.path modification and just drop the _vendor directory. |
As @encukou noted, quite a few of the packages vendored are done so to allow compatibility with older python versions. These will go away soon as we do plan on dropping 2.7 and 3.5 support in a future release. The suggested apprach might work. However, I do need to run this by the other maintainers to make sure I have not missed something obvious, especially since I have not worked on the vendoring here. Will also need to test it out to make sure it does not break anything. Will see if I can get this done before we get to 1.1.0. |
Initial fix for this has been merged into master. For debundling dependencies you can simply drop We'll look into any issues that crop up as they are identified. |
This looks great, thank you! I'll report back if I'm encountering any issues in the packaging context :) |
Just a question: Are those needed to run Poetry on the older Python versions, or also to manage older Python versions projects? E.g. if Poetry itself runs on Python 3.8 to manage a Python 3.6 project, is |
@hroncok with the change I have done the following. As for the remaining bundled dependencies; you will notice that everything except Also, it is important to not that this is specific to Let me know if I can provide more information on this or if something needs to be clarified further. |
Thanks! |
@hroncok I realised I might not have answered your question.
Poetry, when run on 3.8 to manage a 3.6 project, it does not need |
This was resolved via python-poetry/poetry-core#25 |
pip's has a published rationale for doing so: https://pip.pypa.io/en/latest/development/vendoring-policy/#rationale. As noted by others, it's uniquely positioned in this situation. :) |
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Python projects are starting to rely on pyproject.toml / poetry based build systems now, and this is also affecting fedora packages, for example. In this context, it's important to have a working, "native" poetry package.
However, using vendored dependencies - like in this project (which looks like will be required by what will be poetry 1.1.0) - will make it almost impossible to package poetry, let alone package it correctly.
Please reconsider the decision to use bundled dependencies.
The text was updated successfully, but these errors were encountered: