Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc/python: improve description of community tools #75074

Open
wants to merge 1 commit into
base: master
from

Conversation

@ariutta
Copy link
Contributor

@ariutta ariutta commented Dec 6, 2019

Motivation for this change

Added pip2nix, nixpkgs-pytools and poetry2nix, because they are actively maintained and in nix-community. Removed python2nix, because it's been broken since 2017 and is still broken.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Notify maintainers

cc @

Added pip2nix, nixpkgs-pytools and poetry2nix. Removed [python2nix](https://github.com/proger/python2nix), because it's been broken [since 2017](#30887) and is [still broken](proger/python2nix#6).
- [nixpkgs-pytools](https://github.com/nix-community/nixpkgs-pytools): tools for removing the tedious nature of creating nixpkgs derivations.
- [poetry2nix](https://github.com/nix-community/poetry2nix): turn Poetry projects into Nix derivations without the need to actually write Nix expressions.

Note that pypi2nix maintains a separate package tree from nixpkgs, so Python package derivations from pypi2nix and nixpkgs [should not be used together](https://github.com/nix-community/pypi2nix/issues/222#issuecomment-443497376). But Python package derivations from pip2nix or nixpkgs-pytools can be used together with those from nixpkgs.

This comment has been minimized.

@worldofpeace

worldofpeace Dec 6, 2019
Member

The part that's confusing here is it leaves out poetry2nix, can it be used together with nixpkgs?
If it can, can we phrase this like,

should not be used together. The other tools mentioned don't have this limitation.
@ariutta
Copy link
Contributor Author

@ariutta ariutta commented Dec 6, 2019

@worldofpeace
Copy link
Member

@worldofpeace worldofpeace commented Dec 6, 2019

Good idea. I think poetry2nix can be used this way, but I’m not 100% sure. I’m more familiar with the other tools.

On Dec 5, 2019, at 6:48 PM, worldofpeace @.***> wrote:  @worldofpeace requested changes on this pull request. In doc/languages-frameworks/python.section.md: > -- pypi2nix: Generate Nix expressions for your Python project. Note that [sharing derivations from pypi2nix with nixpkgs is possible but not encouraged](nix-community/pypi2nix#222 (comment)). -- python2nix by Vladimir Kirillov. +- pypi2nix: generate Nix expressions for Python packages. +- pip2nix: generate Nix expressions for Python packages. +- nixpkgs-pytools: tools for removing the tedious nature of creating nixpkgs derivations. +- poetry2nix: turn Poetry projects into Nix derivations without the need to actually write Nix expressions. + +Note that pypi2nix maintains a separate package tree from nixpkgs, so Python package derivations from pypi2nix and nixpkgs [should not be used together](nix-community/pypi2nix#222 (comment)). But Python package derivations from pip2nix or nixpkgs-pytools can be used together with those from nixpkgs. The part that's confusing here is it leaves out poetry2nix, can it be used together with nixpkgs? If it can, can we phrase this like, should not be used together. The other tools mentioned don't have this limitation. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

I guess we can ask @adisbladis 👋

@adisbladis
Copy link
Member

@adisbladis adisbladis commented Dec 6, 2019

You can use poetry2nix together with packages from nixpkgs but it should not be required other than a very small number of cases, for example missing inputs like setuptools_scm and the like.

Other than that poetry's lockfile contains the whole environment, so you shouldn't really need to.

The short answer: You can but it's discouraged and should be discouraged for all other tools too because it's unsafe.

@adisbladis
Copy link
Member

@adisbladis adisbladis commented Dec 6, 2019

Note: I took a quick look at how pip2nix generates packages and it's also unsafe to mix nixpkgs python packages.

@ariutta
Copy link
Contributor Author

@ariutta ariutta commented Dec 13, 2019

@datakurre
Copy link
Contributor

@datakurre datakurre commented Dec 13, 2019

@datakurre, could you comment on this pull request regarding documentation on mixing pip2nix python packages with nixpkgs python packages?

@adisbladis I'm also interested in hearing, what makes mixing with nixpkgs' Python package set unsafe? In my understanding it could be very fragile (prone to conflicting paths), but I'm not aware of safety issues.

@ariutta Whenever pip2nix is used to extend nixpkgs' Python packages with a set that includes a package already in nixpkgs, it is very easy to get a result with multiple versions of the same package. That said, I've been trying to maintain pip2nix, because I have been using it for years to develop and build Python-based microservices with my "setup.nix"-project (and most of "setup.nix" is about aligning pip2nix-package set with nixpkgs set to prevent conflicts by duplicates).

Here's an example of my projects that has recently used Nix based Python development environment with reasonable big package sets: https://github.com/robots-from-jupyter/robotkernel

@ariutta
Copy link
Contributor Author

@ariutta ariutta commented Dec 22, 2019

@worldofpeace, the feedback from @adisbladis and @datakurre suggests there is not a clear consensus for how to word the last sentence of my pull request. Would you prefer to just delete that sentence, or should we try to come up with a consensus?

@adisbladis and @datakurre, does the following sound OK?

While nixpkgs and all the tools above are valid sources of Python package derivations, it is safest to pick just one, in order to avoid package tree conflicts. If you need to extend the Python package derivations from nixpkgs, you can write your own or try using setup.nix.

@adisbladis
Copy link
Member

@adisbladis adisbladis commented Dec 22, 2019

Most solutions can possibly have more than one version of a package in a closure.

This can easily happen if something in nixpkgs overrides a dependency in a let-block. That's a big no-no but is relatively commonplace in nixpkgs.

Take the following expression as an example: https://github.com/NixOS/nixpkgs/blob/ac4b42434a52ad4d0b728c9a59acae2a73df755e/pkgs/development/python-modules/poetry/default.nix.
It has an override of cleo

cleo6 = cleo.overridePythonAttrs (oldAttrs: rec {
.
This means that you cannot reasonably know which version of cleo this expression will use at runtime:

{ buildPythonApplication, cleo, poetry }:
buildPythonApplication {
  propagatedBuildInputs = [
      poetry
      cleo
  ];
  ...
}

The only safe way AFAIK to mix anything generated with nixpkgs is through packageOverrides which ensures a coherent python environment.
This is what poetry2nix is using and why it should be safe to pull in build systems and the like from the nixpkgs set despite them not being in poetry.lock, given that they also do not use any local override shenanigans.

I have personally been bitten by this and ended up using an outdated version of requests in production which blew up at runtime despite passing all my tests...

I'm guessing the cause of this is not enough awareness/understanding of these mechanics by maintainers.

@adisbladis
Copy link
Member

@adisbladis adisbladis commented Dec 23, 2019

My recommendation: Either remove any section on using generated python sets together with python packages from nixpkgs or put a big fat blanket warning label that this is something to be avoided.

@worldofpeace
Copy link
Member

@worldofpeace worldofpeace commented Dec 23, 2019

My recommendation: Either remove any section on using generated python sets together with python packages from nixpkgs or put a big fat blanket warning label that this is something to be avoided.

That's also good with me @ariutta, you don't have to be even specific. (maybe an example could be useful to describe what that looks like)

@datakurre
Copy link
Contributor

@datakurre datakurre commented Dec 23, 2019

I can agree with the comments above too 👍

@FRidh FRidh added this to the 20.03 milestone Feb 5, 2020
@FRidh FRidh removed this from the 20.03 milestone Mar 30, 2020
@FRidh FRidh self-assigned this Mar 30, 2020
@stale
Copy link

@stale stale bot commented Sep 26, 2020

Hello, I'm a bot and I thank you in the name of the community for your contributions.

Nixpkgs is a busy repository, and unfortunately sometimes PRs get left behind for too long. Nevertheless, we'd like to help committers reach the PRs that are still important. This PR has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

If this is still important to you and you'd like to remove the stale label, we ask that you leave a comment. Your comment can be as simple as "still important to me". But there's a bit more you can do:

If you received an approval by an unprivileged maintainer and you are just waiting for a merge, you can @ mention someone with merge permissions and ask them to help. You might be able to find someone relevant by using Git blame on the relevant files, or via GitHub's web interface. You can see if someone's a member of the nixpkgs-committers team, by hovering with the mouse over their username on the web interface, or by searching them directly on the list.

If your PR wasn't reviewed at all, it might help to find someone who's perhaps a user of the package or module you are changing, or alternatively, ask once more for a review by the maintainer of the package/module this is about. If you don't know any, you can use Git blame on the relevant files, or GitHub's web interface to find someone who touched the relevant files in the past.

If your PR has had reviews and nevertheless got stale, make sure you've responded to all of the reviewer's requests / questions. Usually when PR authors show responsibility and dedication, reviewers (privileged or not) show dedication as well. If you've pushed a change, it's possible the reviewer wasn't notified about your push via email, so you can always officially request them for a review, or just @ mention them and say you've addressed their comments.

Lastly, you can always ask for help at our Discourse Forum, or more specifically, at this thread or at #nixos' IRC channel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants
You can’t perform that action at this time.