Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

poetry2nix: init at 1.1.0 #76646

Merged
merged 2 commits into from Jan 2, 2020
Merged

poetry2nix: init at 1.1.0 #76646

merged 2 commits into from Jan 2, 2020

Conversation

@adisbladis
Copy link
Member

@adisbladis adisbladis commented Dec 29, 2019

Motivation for this change

This adds poetry2nix to nixpkgs and uses it to re-package poetry.

This new poetry derivation has a number of advantages over the old one:

  • "Vendored" dependencies
    Prevents clashes between nixpkgs poetry and projects dependencies
  • Uses poetry's own lock files
    This ensures a well-tested set of dependencies.
  • Should be much easier to keep up-to-date
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Notify maintainers

cc @zimbatm @gilligan @andir

@adisbladis adisbladis requested review from FRidh and jonringer as code owners Dec 29, 2019
@adisbladis
Copy link
Member Author

@adisbladis adisbladis commented Dec 29, 2019

@GrahamcOfBorg build poetry

@FRidh
Copy link
Member

@FRidh FRidh commented Dec 29, 2019

While I think it is great for developers to be able to use Poetry in Nix for creating reproducible environments and applications (I will be using it myself), I am a bit skeptical towards using it in Nixpkgs.

Lock files make it quick and easy to package exactly what upstream intended, however, that's not always good. For example, they pin patch versions which means distributing security updates becomes harder. That, along with reporting to upstream issues we encounter is I think an important role of distributions. Clearly, this remark isn't just applicable to poetry2nix but also other such converters as well as Docker images.

@adisbladis
Copy link
Member Author

@adisbladis adisbladis commented Dec 31, 2019

Lock files make it quick and easy to package exactly what upstream intended, however, that's not always good. For example, they pin patch versions which means distributing security updates becomes harder.

While this is true there is nothing that says we need to use uptreams lock files, throwing away upstream's lock-file and running poetry lock is imho perfectly fine.
There is also the question of correctness, it's currently really hard to ensure that bumping a python package does not cause breakage elsewhere, having per-application environments fixes that problem at it's root.

Another thing about this: There are a lot of applications out there with complex dependency graphs that I will never package in nixpkgs without poetry2nix.
I'm simply not interested in spending time on packaging pretix or pretalx without proper automation.

@adisbladis adisbladis force-pushed the adisbladis:poetry branch 2 times, most recently from 69f7607 to 1f40362 Dec 31, 2019
@ofborg ofborg bot removed the 6.topic: python label Dec 31, 2019
@adisbladis adisbladis force-pushed the adisbladis:poetry branch from 1f40362 to 3ed94c7 Dec 31, 2019
@zimbatm
Copy link
Member

@zimbatm zimbatm commented Jan 1, 2020

We could also use poetry2nix to maintain the current python packages set.

The more we can move towards pure data in nixpkgs, the more it will allow us to write higher-level tooling to automate all the grunt work. In the future, GitHub will be mostly used by bots talking to each-other while we sip margaritas at the beach (or do other more interesting things than manual dependency resolution).

@FRidh
Copy link
Member

@FRidh FRidh commented Jan 1, 2020

While this is true there is nothing that says we need to use uptreams lock files, throwing away upstream's lock-file and running poetry lock is imho perfectly fine.

This is a good idea I think. So long-term we need a way to regenerate (all) lock files and make them use the latest patch releases.

There are a lot of applications out there with complex dependency graphs that I will never package in nixpkgs without poetry2nix.

Clearly. For these kinds of applications it just does not make sense to use only the main packages set.

We could also use poetry2nix to maintain the current python packages set.

We could start a thread about this on Discourse but I can tell you already now it will fail with version conflicts when resolving dependencies (locking). Like now a way is needed to handle such situation.

@adisbladis adisbladis force-pushed the adisbladis:poetry branch from 3ed94c7 to f500f94 Jan 1, 2020
@ofborg ofborg bot requested a review from jbaum98 Jan 1, 2020
@adisbladis adisbladis force-pushed the adisbladis:poetry branch 4 times, most recently from a6da5d6 to 6195894 Jan 1, 2020
@adisbladis adisbladis changed the title WIP: poetry2nix: init at 1.0.0 poetry2nix: init at 1.0.0 Jan 1, 2020
@adisbladis adisbladis force-pushed the adisbladis:poetry branch from 6195894 to 39d163b Jan 2, 2020
@adisbladis
Copy link
Member Author

@adisbladis adisbladis commented Jan 2, 2020

I consider this merge-ready.

@adisbladis adisbladis force-pushed the adisbladis:poetry branch from 39d163b to d009f23 Jan 2, 2020
@FRidh
FRidh approved these changes Jan 2, 2020
@adisbladis adisbladis force-pushed the adisbladis:poetry branch from d009f23 to b0ba3f9 Jan 2, 2020
@adisbladis
Copy link
Member Author

@adisbladis adisbladis commented Jan 2, 2020

@GrahamcOfBorg build poetry

adisbladis added 2 commits Dec 29, 2019
@adisbladis adisbladis force-pushed the adisbladis:poetry branch from b0ba3f9 to 54928f9 Jan 2, 2020
@adisbladis adisbladis changed the title poetry2nix: init at 1.0.0 poetry2nix: init at 1.1.0 Jan 2, 2020
@adisbladis adisbladis merged commit 1b33a0a into NixOS:master Jan 2, 2020
2 of 4 checks passed
2 of 4 checks passed
grahamcofborg-eval Beginning Evaluations
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release-combined.nix -A tested
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
@adisbladis adisbladis mentioned this pull request Jan 2, 2020
3 of 10 tasks complete
@knedlsepp
Copy link
Member

@knedlsepp knedlsepp commented Jan 3, 2020

This is great! Makes me want to introduce poetry in all my python projects. Is there any chance however that this did break python36Packages.pyopenssl?

@adisbladis
Copy link
Member Author

@adisbladis adisbladis commented Jan 3, 2020

@knedlsepp Nope, nothing in pythonPackages was touched in this PR.

@knedlsepp
Copy link
Member

@knedlsepp knedlsepp commented Jan 3, 2020

@adisbladis: Oh. Sorry about that, I only noticed that it did break somewhere between 62ef77...ec29bb and this commit at least has pyopenssl in the overrides.nix. Verified that this is unrelated.
(Edit: Turns out pyopenssl doesn't build locally for 62ef77 either, hydra did succeed for some reason though.)

@asymmetric asymmetric mentioned this pull request Jan 4, 2020
1 of 1 task complete
@asymmetric
Copy link
Contributor

@asymmetric asymmetric commented Jan 4, 2020

One of our macOS Travis jobs seems to be broken since this got merged.

The error is:

$ nix-env -i ShellCheck
grep -RHL --exclude-dir node_modules node src/*/libexec | xargs shellcheck --exclude=2001

error: attribute 'fromTOML' missing, at /nix/store/6vni58c49p2pwmpzkjr7zsf1b2arsd98-nixpkgs-20.03pre207234.ec29bb50bf4/nixpkgs/pkgs/development/tools/poetry2nix/poetry2nix/lib.nix:27:20

(use '--show-trace' to show detailed location information)
@adisbladis
Copy link
Member Author

@adisbladis adisbladis commented Jan 4, 2020

@asymmetric I suggest upgrading your nix version to 2.1 or newer.

@zimbatm
Copy link
Member

@zimbatm zimbatm commented Jan 4, 2020

Why is poetry2nix being evaluated at all? That sounds bad for performance..

@zimbatm
Copy link
Member

@zimbatm zimbatm commented Jan 4, 2020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

6 participants
You can’t perform that action at this time.