Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pythonPackages: new pypi2nix implementation #1903

Closed

Conversation

@offlinehacker
Copy link
Contributor

@offlinehacker offlinehacker commented Mar 8, 2014

The new pypi2nix implementation (maybe the last one :P), do not merge it yet!

Project page: https://github.com/offlinehacker/pypi2nix

This will be released soon, but currently it's up to nix/python developers to test it and report any problems. Also buildout versions parsing support has to be added asoap( @garbas ?). There's also one new file python-overrrides.nix where you speciffy nix overrides for generated packages. For additional documentation please reffer to project page.

To install:

  • merge/cherry-pick this branch
  • install pythonPackages.pypi2nix

To test:

  • run: pypi2nix pkgs/top-level/python-packages.json pkgs/top-level/python-packages-generated.nix
  • update: pypi2nix --update pkgs/top-level/python-packages.json pkgs/top-level/python-packages-generated.nix

@garbas @iElectric @matejc you might be intereseted.

Thanks for all the support :)

@offlinehacker
Copy link
Contributor Author

@offlinehacker offlinehacker commented Mar 9, 2014

I made an update

@garbas
garbas reviewed Mar 9, 2014
View changes
pkgs/top-level/python-packages.json Outdated
" s|^TIFF_ROOT =.*$|TIFF_ROOT = _lib_include(\"${pkgs.libtiff}\")|g ;'"
]
],
"overrides": {

This comment has been minimized.

@garbas

garbas Mar 9, 2014
Contributor

what is the reason for "overrides" section? could we just have "deps", "requirements", ... with the package above?

@domenkozar
Copy link
Member

@domenkozar domenkozar commented Mar 9, 2014

Putting versions in names won't work. It kills nix ability to handle updates. Instead I'd suggest each package has an alias if multiple packages need to be supported. For example alias: "django_1_4"

@offlinehacker
Copy link
Contributor Author

@offlinehacker offlinehacker commented Mar 9, 2014

@iElectric, it already works that way :)

On Sun, Mar 9, 2014 at 8:41 PM, Domen Kožar notifications@github.comwrote:

Putting versions in names won't work. It kills nix ability to handle
updates. Instead I'd suggest each package has an alias if multiple packages
need to be supported. For example alias: "django_1_4"


Reply to this email directly or view it on GitHubhttps://github.com//pull/1903#issuecomment-37136510
.

@offlinehacker
Copy link
Contributor Author

@offlinehacker offlinehacker commented Mar 9, 2014

@garbas, in practice no, sometimes you have to override in genration phase,
to have everything package managed. For example in python3 you have to pick
unittest2py3k instead of unittest2. If i wouldn't do this, the source would
have to be patched later and then it would lose all the point, because it's
up to pypi2nix to update all the versions. There are also cases where you
have to inject python dependencies, because they are missing. It's not
perfect, but pypi2nix will handle all of this ;)

On Sun, Mar 9, 2014 at 9:56 PM, Jaka Hudoklin jakahudoklin@gmail.comwrote:

@iElectric, it already works that way :)

On Sun, Mar 9, 2014 at 8:41 PM, Domen Kožar notifications@github.comwrote:

Putting versions in names won't work. It kills nix ability to handle
updates. Instead I'd suggest each package has an alias if multiple packages
need to be supported. For example alias: "django_1_4"


Reply to this email directly or view it on GitHubhttps://github.com//pull/1903#issuecomment-37136510
.

@offlinehacker
Copy link
Contributor Author

@offlinehacker offlinehacker commented Mar 9, 2014

@garbas, i also have to implement, per package overrides, now there are
only global, per environment overrides and this might not be enough in some
cases.

On Sun, Mar 9, 2014 at 10:00 PM, Jaka Hudoklin jakahudoklin@gmail.comwrote:

@garbas, in practice no, sometimes you have to override in genration
phase, to have everything package managed. For example in python3 you have
to pick unittest2py3k instead of unittest2. If i wouldn't do this, the
source would have to be patched later and then it would lose all the point,
because it's up to pypi2nix to update all the versions. There are also
cases where you have to inject python dependencies, because they are
missing. It's not perfect, but pypi2nix will handle all of this ;)

On Sun, Mar 9, 2014 at 9:56 PM, Jaka Hudoklin jakahudoklin@gmail.comwrote:

@iElectric, it already works that way :)

On Sun, Mar 9, 2014 at 8:41 PM, Domen Kožar notifications@github.comwrote:

Putting versions in names won't work. It kills nix ability to handle
updates. Instead I'd suggest each package has an alias if multiple packages
need to be supported. For example alias: "django_1_4"


Reply to this email directly or view it on GitHubhttps://github.com//pull/1903#issuecomment-37136510
.

@offlinehacker
Copy link
Contributor Author

@offlinehacker offlinehacker commented Mar 9, 2014

@iElectric: to exlain you better. It's actually really smart. If only name
is passed it extracts spec and alias from it. If you pass name and spec,
name is used for alias and spec is used for package search. As you can see
you can also have per environemnt specs, because that's what you need in
python.

On Sun, Mar 9, 2014 at 10:03 PM, Jaka Hudoklin jakahudoklin@gmail.comwrote:

@garbas, i also have to implement, per package overrides, now there are
only global, per environment overrides and this might not be enough in some
cases.

On Sun, Mar 9, 2014 at 10:00 PM, Jaka Hudoklin jakahudoklin@gmail.comwrote:

@garbas, in practice no, sometimes you have to override in genration
phase, to have everything package managed. For example in python3 you have
to pick unittest2py3k instead of unittest2. If i wouldn't do this, the
source would have to be patched later and then it would lose all the point,
because it's up to pypi2nix to update all the versions. There are also
cases where you have to inject python dependencies, because they are
missing. It's not perfect, but pypi2nix will handle all of this ;)

On Sun, Mar 9, 2014 at 9:56 PM, Jaka Hudoklin jakahudoklin@gmail.comwrote:

@iElectric, it already works that way :)

On Sun, Mar 9, 2014 at 8:41 PM, Domen Kožar notifications@github.comwrote:

Putting versions in names won't work. It kills nix ability to handle
updates. Instead I'd suggest each package has an alias if multiple packages
need to be supported. For example alias: "django_1_4"


Reply to this email directly or view it on GitHubhttps://github.com//pull/1903#issuecomment-37136510
.

@offlinehacker
Copy link
Contributor Author

@offlinehacker offlinehacker commented Mar 9, 2014

Expect more updates to pypi2nix in following days. The goal will be reached when most of the packages we have are managed by pypi2nix, then we will merge.

@offlinehacker
Copy link
Contributor Author

@offlinehacker offlinehacker commented Mar 10, 2014

Me, @iElectric and @garbas have discussed the problems that has to be additionaly solved and defined updated source format for generating. @garbas also came up with an append like solution, which would not require downloading of all packages from pypi when you are just adding a simple package and an update of all packages is not needed. Hopefully this will finished in a few weeks.

@shlevy
Copy link
Member

@shlevy shlevy commented Mar 30, 2014

@offlinehacker
Copy link
Contributor Author

@offlinehacker offlinehacker commented Mar 30, 2014

It's not real solution, since it does not do dependency handling. It's just
a helper tool that copies metadata.
On Mar 9, 2014 8:41 PM, "Domen Kožar" notifications@github.com wrote:

Putting versions in names won't work. It kills nix ability to handle
updates. Instead I'd suggest each package has an alias if multiple packages
need to be supported. For example alias: "django_1_4"


Reply to this email directly or view it on GitHubhttps://github.com//pull/1903#issuecomment-37136510
.

@offlinehacker
Copy link
Contributor Author

@offlinehacker offlinehacker commented Mar 30, 2014

Currently i'm waiting for garbas to refract input json format parser, i
have a few more bugs, then i will try to package most of the packages we
currently have with pypi2nix
On Mar 30, 2014 6:08 PM, "Jaka Hudoklin" jakahudoklin@gmail.com wrote:

It's not real solution, since it does not do dependency handling. It's
just a helper tool that copies metadata.
On Mar 9, 2014 8:41 PM, "Domen Kožar" notifications@github.com wrote:

Putting versions in names won't work. It kills nix ability to handle
updates. Instead I'd suggest each package has an alias if multiple packages
need to be supported. For example alias: "django_1_4"


Reply to this email directly or view it on GitHubhttps://github.com//pull/1903#issuecomment-37136510
.

@domenkozar
Copy link
Member

@domenkozar domenkozar commented Mar 31, 2014

It's perfect for what we have now. Generates a template with given information except for dependencies. Will give it a try.

@offlinehacker
Copy link
Contributor Author

@offlinehacker offlinehacker commented Mar 31, 2014

You could also try pypi2nix, it does all that and more for whole dependency
tree including testing dependencies.
On Mar 31, 2014 4:48 PM, "Domen Kožar" notifications@github.com wrote:

It's perfect for what we have now. Generates a template with given
information except for dependencies. Will give it a try.


Reply to this email directly or view it on GitHubhttps://github.com//pull/1903#issuecomment-39096852
.

@shlevy
Copy link
Member

@shlevy shlevy commented Apr 1, 2014

When I last asked I was told pypi2nix wasn't usable yet, has that changed?

@proger
Copy link
Contributor

@proger proger commented Apr 1, 2014

@iElectric it actually generates a list of dependencies, however it doesn't render any expressions for dependencies itself unless explicitly asked for (because most likely they are already in python-packages.nix)

regarding pypi2nix — why we need to write json files instead of json expressions? and how does it handle native dependencies?

@offlinehacker
Copy link
Contributor Author

@offlinehacker offlinehacker commented Apr 30, 2014

@garbas, @iElectric and also others can you check out new refactored verison of pypi2nix: https://github.com/offlinehacker/pypi2nix (yes new format is supported)

Currently it packages big packages like sentry, celery and similar without of problems(including tests). Of course you need to do overrides(also in generation phase), because a quality of python package managment sucks, but my pypi2nix supports it, so you can do any kind of kung-fu with packages, like adding dependencies, removing them, changing them, changing source,... in generation phase and later in nix. I use it in production on one of projects, it works great :) Especially look into examples folder for example input specs.

@garbas can you add buildout version parsing support, there's a function mockup already :)

It still needs some works, like support for differential generation, better test coverage, detection and repair of dependency cycles, bulidout support, but besides that i think it works great :)

@offlinehacker
Copy link
Contributor Author

@offlinehacker offlinehacker commented Apr 30, 2014

That's how i use it in one of my projects: https://gist.github.com/offlinehacker/0639613e23f508aa1572

Run it with: pypi2nix --enabled-envs python27 --verbose --extra test,develop . generated.nix

@garbas
Copy link
Contributor

@garbas garbas commented Apr 30, 2014

@offlinehacker awesome! i have no time for until monday, but i want to start using this first thing next week. i will implement that buildout versions parsing asap and test it with plone. then we have pypi2nix working for all packages that was working before (just better). again ... awesome work.

@offlinehacker
Copy link
Contributor Author

@offlinehacker offlinehacker commented May 1, 2014

@garbas thanks! Ultimate goal is to try to generate packages for whole pypi, and do some statistics of quality of python packages and write some cool article.

@offlinehacker
Copy link
Contributor Author

@offlinehacker offlinehacker commented May 28, 2014

@garbas @iElectric I just did an update with a lot more packages and fully functional pypi2nix :) Still waiting for buildout support.

@domenkozar
Copy link
Member

@domenkozar domenkozar commented Aug 25, 2014

Closing until the PR is ready for merge.

@domenkozar domenkozar closed this Aug 25, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
You can’t perform that action at this time.