pythonPackages: new pypi2nix implementation

[offlinehacker]

The new pypi2nix implementation (maybe the last one :P), do not merge it yet!

Project page: https://github.com/offlinehacker/pypi2nix

This will be released soon, but currently it's up to nix/python developers to test it and report any problems. Also buildout versions parsing support has to be added asoap( @garbas ?). There's also one new file python-overrrides.nix where you speciffy nix overrides for generated packages. For additional documentation please reffer to project page.

To install:

• merge/cherry-pick this branch
• install pythonPackages.pypi2nix

To test:

• run: pypi2nix pkgs/top-level/python-packages.json pkgs/top-level/python-packages-generated.nix
• update: pypi2nix --update pkgs/top-level/python-packages.json pkgs/top-level/python-packages-generated.nix

@garbas @iElectric @matejc you might be intereseted.

Thanks for all the support :)

[offlinehacker]

I made an update

[garbas]

what is the reason for "overrides" section? could we just have "deps", "requirements", ... with the package above?

[domenkozar]

Putting versions in names won't work. It kills nix ability to handle updates. Instead I'd suggest each package has an alias if multiple packages need to be supported. For example alias: "django_1_4"

[offlinehacker]

@iElectric, it already works that way :)

On Sun, Mar 9, 2014 at 8:41 PM, Domen Kožar notifications@github.comwrote:

Putting versions in names won't work. It kills nix ability to handle
updates. Instead I'd suggest each package has an alias if multiple packages
need to be supported. For example alias: "django_1_4"

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1903#issuecomment-37136510
.

[offlinehacker]

@garbas, in practice no, sometimes you have to override in genration phase,
to have everything package managed. For example in python3 you have to pick
unittest2py3k instead of unittest2. If i wouldn't do this, the source would
have to be patched later and then it would lose all the point, because it's
up to pypi2nix to update all the versions. There are also cases where you
have to inject python dependencies, because they are missing. It's not
perfect, but pypi2nix will handle all of this ;)

On Sun, Mar 9, 2014 at 9:56 PM, Jaka Hudoklin jakahudoklin@gmail.comwrote:

@iElectric, it already works that way :)

On Sun, Mar 9, 2014 at 8:41 PM, Domen Kožar notifications@github.comwrote:

Putting versions in names won't work. It kills nix ability to handle
updates. Instead I'd suggest each package has an alias if multiple packages
need to be supported. For example alias: "django_1_4"

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1903#issuecomment-37136510
.

[offlinehacker]

@garbas, i also have to implement, per package overrides, now there are
only global, per environment overrides and this might not be enough in some
cases.

On Sun, Mar 9, 2014 at 10:00 PM, Jaka Hudoklin jakahudoklin@gmail.comwrote:

@garbas, in practice no, sometimes you have to override in genration
phase, to have everything package managed. For example in python3 you have
to pick unittest2py3k instead of unittest2. If i wouldn't do this, the
source would have to be patched later and then it would lose all the point,
because it's up to pypi2nix to update all the versions. There are also
cases where you have to inject python dependencies, because they are
missing. It's not perfect, but pypi2nix will handle all of this ;)

On Sun, Mar 9, 2014 at 9:56 PM, Jaka Hudoklin jakahudoklin@gmail.comwrote:

@iElectric, it already works that way :)

On Sun, Mar 9, 2014 at 8:41 PM, Domen Kožar notifications@github.comwrote:

Putting versions in names won't work. It kills nix ability to handle
updates. Instead I'd suggest each package has an alias if multiple packages
need to be supported. For example alias: "django_1_4"

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1903#issuecomment-37136510
.

[offlinehacker]

@iElectric: to exlain you better. It's actually really smart. If only name
is passed it extracts spec and alias from it. If you pass name and spec,
name is used for alias and spec is used for package search. As you can see
you can also have per environemnt specs, because that's what you need in
python.

On Sun, Mar 9, 2014 at 10:03 PM, Jaka Hudoklin jakahudoklin@gmail.comwrote:

@garbas, i also have to implement, per package overrides, now there are
only global, per environment overrides and this might not be enough in some
cases.

On Sun, Mar 9, 2014 at 10:00 PM, Jaka Hudoklin jakahudoklin@gmail.comwrote:

@garbas, in practice no, sometimes you have to override in genration
phase, to have everything package managed. For example in python3 you have
to pick unittest2py3k instead of unittest2. If i wouldn't do this, the
source would have to be patched later and then it would lose all the point,
because it's up to pypi2nix to update all the versions. There are also
cases where you have to inject python dependencies, because they are
missing. It's not perfect, but pypi2nix will handle all of this ;)

On Sun, Mar 9, 2014 at 9:56 PM, Jaka Hudoklin jakahudoklin@gmail.comwrote:

@iElectric, it already works that way :)

On Sun, Mar 9, 2014 at 8:41 PM, Domen Kožar notifications@github.comwrote:

Putting versions in names won't work. It kills nix ability to handle
updates. Instead I'd suggest each package has an alias if multiple packages
need to be supported. For example alias: "django_1_4"

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1903#issuecomment-37136510
.

[offlinehacker]

Expect more updates to pypi2nix in following days. The goal will be reached when most of the packages we have are managed by pypi2nix, then we will merge.

[offlinehacker]

Me, @iElectric and @garbas have discussed the problems that has to be additionaly solved and defined updated source format for generating. @garbas also came up with an append like solution, which would not require downloading of all packages from pypi when you are just adding a simple package and an update of all packages is not needed. Hopefully this will finished in a few weeks.

[shlevy]

@proger's https://github.com/proger/python2nix might be of interest.

[offlinehacker]

It's not real solution, since it does not do dependency handling. It's just
a helper tool that copies metadata.
On Mar 9, 2014 8:41 PM, "Domen Kožar" notifications@github.com wrote:

Putting versions in names won't work. It kills nix ability to handle
updates. Instead I'd suggest each package has an alias if multiple packages
need to be supported. For example alias: "django_1_4"

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1903#issuecomment-37136510
.

[offlinehacker]

Currently i'm waiting for garbas to refract input json format parser, i
have a few more bugs, then i will try to package most of the packages we
currently have with pypi2nix
On Mar 30, 2014 6:08 PM, "Jaka Hudoklin" jakahudoklin@gmail.com wrote:

It's not real solution, since it does not do dependency handling. It's
just a helper tool that copies metadata.
On Mar 9, 2014 8:41 PM, "Domen Kožar" notifications@github.com wrote:

Putting versions in names won't work. It kills nix ability to handle
updates. Instead I'd suggest each package has an alias if multiple packages
need to be supported. For example alias: "django_1_4"

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1903#issuecomment-37136510
.

[domenkozar]

It's perfect for what we have now. Generates a template with given information except for dependencies. Will give it a try.

[offlinehacker]

You could also try pypi2nix, it does all that and more for whole dependency
tree including testing dependencies.
On Mar 31, 2014 4:48 PM, "Domen Kožar" notifications@github.com wrote:

It's perfect for what we have now. Generates a template with given
information except for dependencies. Will give it a try.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1903#issuecomment-39096852
.

[shlevy]

When I last asked I was told pypi2nix wasn't usable yet, has that changed?

[proger]

@iElectric it actually generates a list of dependencies, however it doesn't render any expressions for dependencies itself unless explicitly asked for (because most likely they are already in python-packages.nix)

regarding pypi2nix — why we need to write json files instead of json expressions? and how does it handle native dependencies?

[offlinehacker]

@garbas, @iElectric and also others can you check out new refactored verison of pypi2nix: https://github.com/offlinehacker/pypi2nix (yes new format is supported)

Currently it packages big packages like sentry, celery and similar without of problems(including tests). Of course you need to do overrides(also in generation phase), because a quality of python package managment sucks, but my pypi2nix supports it, so you can do any kind of kung-fu with packages, like adding dependencies, removing them, changing them, changing source,... in generation phase and later in nix. I use it in production on one of projects, it works great :) Especially look into examples folder for example input specs.

@garbas can you add buildout version parsing support, there's a function mockup already :)

It still needs some works, like support for differential generation, better test coverage, detection and repair of dependency cycles, bulidout support, but besides that i think it works great :)

[offlinehacker]

That's how i use it in one of my projects: https://gist.github.com/offlinehacker/0639613e23f508aa1572

Run it with: pypi2nix --enabled-envs python27 --verbose --extra test,develop . generated.nix

[garbas]

@offlinehacker awesome! i have no time for until monday, but i want to start using this first thing next week. i will implement that buildout versions parsing asap and test it with plone. then we have pypi2nix working for all packages that was working before (just better). again ... awesome work.

[offlinehacker]

@garbas thanks! Ultimate goal is to try to generate packages for whole pypi, and do some statistics of quality of python packages and write some cool article.

[offlinehacker]

@garbas @iElectric I just did an update with a lot more packages and fully functional pypi2nix :) Still waiting for buildout support.

[domenkozar]

Closing until the PR is ready for merge.