Python: clean up old versions of packages

[FRidh]

Issue description

Occasionally we add multiple versions of Python libraries because certain packages cannot use the current/latest version. Sometimes these older versions are left behind or upgrading to the latest version wasn't attempted. Let's check whether we still need all these old versions.

The following is a list of attributes that contain a version (thanks to @orivej) and that could be older versions.

• amqp_1. Needed for kombu_3 which is needed for oslo which isn't maintained.
• click_5. Needed for platformio. @mogorman (note that your handle doesn't match your GH name)
• kombu_3. See amgp_1.
• beautifulsoup4. Upstream name.
• boto3. Upstream name
• deform2. Duplicate 63550e3
• gmpy2. Upstream name.
• pycurl2. Upstream name.
• pyparsing1. Used in mwlib. 6c467b7
• pyro3. Upstream name is Pyro. Package is only used by openopc f6dd52c
• requests2. We should rename this to requests and see whether we can get rid of the current requests. f63eb58
• tqdm4. a33f9b0
• beautifulsoup_4_1_3. Only used by flexget. See flexget: 1.2.337 -> 2.8.17 #21636 c1cbae5
• dateutil_2_2 Only used by csvkit. @vrthra perhaps upgrade csvkit? csvkit: 0.9.1 -> 1.0.2 #25404
• dateutil_1_5. Used by couple older packages.
• dateutil_2_1. Only needed by flexget flexget: 1.2.337 -> 2.8.17 #21636 c1cbae5
• pathspec 0.3.4: Only used by pants. @copumpkin does pants really need this version? Looking at its setup.py it is very conservative with its dependencies. afad430
• django_1_9. @lsix I know this has been brought up before, but which versions are we going to keep again?
• django_1_8
• django_1_6
• lxml_3_5. Not used at all so lets get rid of it. d3cebfb
• mccabe_0_5. Not used at all so lets get rid of it. 12b2fbf
• openpyxl_2_2_0_b1. @vrthra another dependency of csvkit. csvkit: 0.9.1 -> 1.0.2 #25404
• pymongo_2_9_1. Used by eve and flask-pymongo. Both expressions don't seem to be maintained.
• sqlalchemy8. Needed for two packages but its been broken for a long time. Let's remove it. 1376c66
• sqlalchemy_1_0. This will be the only version left.
• zc_buildout2 @garbas which versions do we need? 63993e7
• zc_buildout221
• zc_buildout171
• zc_buildout152
• billiard_33. Not used. Let's remove it. 4847ef5
• llfuse-0-41. Only used for attic. @bjornfor attic isn't maintained. Are we going to keep the expressions? e750961
• boto-230. Only used by gsutil which is an application with strict requirements. gsutil should probably be moved out of python-packages.nix along with its specific requirements. 7491bc9

[FRidh]

@Denommus you've added the eve package in #12237. Could we get a eve upgrade so we can get rid of the older version of pymongo?

[bjornfor]

We can remove attic. Perhaps set meta.broken now and remove after next release?

[FRidh]

@bjornfor do you think we have attic users that would choose to keep using this backup software even when its unmaintained and marked as broken? I highly doubt they would so I think we can just remove it right away.

[FRidh]

@domenkozar @cillianderoiste does any of you still use almir? It depends on the broken sqlalchemy8 that I like to remove.

[domenkozar]

It doesn't build with sqlalchemy9?

[bjornfor]

@FRidh: I have no idea if there are attic users left, so I thought a deprecation period could be helpful (in case there are). I myself have moved to borgbackup, so for me personally you can remove it right now.

[orivej]

I guess it could help to collect per derivation download statistics from cache.nixos.org, if possible.

[FRidh]

@domenkozar I couldn't get it to work with the latest version. Anyway, because it wasn't working anymore I decided to mark it as broken.

[domenkozar]

Sounds good.

[lsix]

Requarding Django : the ideal solution would be to only maintain Django-1.11, which is the last LTS release (since 1 month).

• Django 1.9 is officially not maintained since april so it would make sense to drop it.
• Django 1.6 is officially not maintained since april 2015 so we should drop it.
• Django 1.10 will be maintained until the end of the year. It would be preferable to not have it in 17.09 since it will not be maintained for entire lifetime of this nixos release.
• Django 1.8 will be maintained upstream until the end of 2018 so there is no objection keeping it.

I’ll have a look at the various versions that could be dropped, bet they have all been kept because they are referenced by other derivations. I’ll review this ASAP.

[7c6f434c]

- Django 1.9 is officially not maintained since april so it would make sense to drop it. - Django 1.8 will be maintained upstream until the end of 2018 so there is no objection keeping it.

django_hijack currently uses 1.9 (I don't use it, but hit an evaluation problem). Is there an update for it, or should it use 1.8?

[FRidh]

@7c6f434c django packages should always depend on django, not on a specific version. When one then wants to use the packages with a specific version of django they can override the django attribute

[lsix]

@7c6f434c django_hijack supports django 1.10. I was just checking if the current release supports django 1-11, but apparently not. We will have to wait until upstream brings support (or do it ourself).

I also saw that the PR I had for django-1.11 brings the evaluation problem. I am checking what I can do about that.

[7c6f434c]

@lsix I think the evaluation and even build part is done now.

@FRidh the package in question doesn't support the default django at all.

[FRidh]

@7c6f434c maybe add a note? If we start using specific versions of django we often end up with multiple versions in a closure.

[7c6f434c]

`django_hijack` seems to be pretty good at failing when its closure contains multiple versions.

[garbas]

I'm not sure who is using zcbuildout I'm not using it. I guess removing and keeping the latest would be a safe way to go forward.

[FRidh]

@datakurre @chaoflow @cillianderoiste you were the last ones contributing to the zc.buildout expressions. Can we remove all but the latest versions?

[datakurre]

@FRidh Keeping the latest version (and the _nix-variant) would be ok for me.

[copumpkin]

@FRidh for pants, yes unfortunately the old pants will build fine with a newer pathspec but it'll fail at runtime. The moment they release 0.13 final (so we can get rid of the stable version) we can kill the old pathspec.

[copumpkin]

@FRidh what I'll do is move (as I said elsewhere) pants to its own file and then only keep the old pathspec in there, rather than in the global package set.

[wizzup]

Will you cleanup pkgs/top-level/python-packages.nix too?

• Expressions for Python libraries are supposed to be in pkgs/development/python-modules/<name>/default.nix.
• buildPythonPackage should be replaced by callPackage

Which one of following should be use ?

• mypackage = callPackage ../development/python-modules/mypackage.nix { };
• mypackage = callPackage ../development/python-modules/mypackage/default.nix { };

I would love to help on this, should I make PR incrementally while working on some packages and refer to this issue?

[FRidh]

That should be

• mypackage = callPackage ../development/python-modules/mypackage/default.nix { };.

Please, go ahead. A commit per package and indeed a reference here would be helpful.

[FRidh]

Closing this issue because basically all older packages have been (re)moved.