Openstack Keystone from Liberty to Ocata

[nlewo]

I started to upgrade Openstack Keystone form Liberty (end of life
since 2016-11-17) to Ocata (latest stable release). I create this
issue to centralize PRs and discussions related to this upgrade.

Keytone Ocata builds successfully and Keystone module tests are passed. I
still have to improve some packages... and submit them. I pushed a
temporary branch in my nixpkgs fork https://github.com/nlewo/nixpkgs/tree/keystone-ocata.

Version of modules are based on https://github.com/openstack/requirements/blob/stable/ocata/upper-constraints.txt.

Some questions/remarks:

• I use pypi2nix to generate nix python packages skeletons. I need a
  feedback for one of them in order to know if the generated format
  is correct (mainly regarding coding style).
  An example is the PR pythonPackages.rfc3986: 0.2.2 -> 0.4.1 #25751

• A lot of dependencies needs to be updated. This will temporary
  breaks other openstack components (Neutron, Nova,...). Can we
  just mark them as broken?

• I needed to add specific version of already existing packages. In
  particular, the version 2.12.5 of pythonPackages.requests. This is
  required by oslo.policy: oslo.policy requires
  requests!=2.12.2,!=2.13.0,>=2.10.0. But to avoid conflict, I also
  add to use this modules version as a dependency of some other
  packages, such as sphinx. I then introduce a new version of
  sphinx. I added the suffix _openstack-ocata to all of this kind of
  packages. Is it ok to add these kind of dependencies?

• What about python version? At this moment, I only used python
  2.7. I don't plan to run Keystone with python 3.x so I don't need
  its dependencies to work with python3.x.
  Could I disable them for python3?

• Python package hacking needs really strict dependencies. I had
  to add them especially for this package.

• Several checkPhase are disabled because there is still a circular
  import problem that I plan to solve.

I plan to also upgrade Nova, Neutron and Glance to the Ocata release.

[hyphon81]

Hello.
I agree to upgrade OpenStack packages.
However, surely, there are many dependency and upgrading may break other packages dependency.
I made a nix code for build OpenStack Newton. Then I had to modify python packages that are depended OpenStack packages at least 137. https://github.com/hyphon81/Nixtack/blob/master/openstack/python-packages.nix
Upgrading to Ocata, it will need to look packages from this page and those dependencies.
https://releases.openstack.org/ocata/

About running OpenStack with python3, I worry too, now.
Reffer from this page, some OpenStack application packages passed test with python3 and now voting. https://wiki.openstack.org/wiki/Python3

[nlewo]

@hyphon81 Happy to see other people interested by these packages, and I also use them to build a kind of devstack. The branch https://github.com/nlewo/nixpkgs/tree/keystone-ocata contains a working Keystone Ocata.

Regarding dependencies, I think https://github.com/openstack/requirements/blob/stable/ocata/upper-constraints.txt. is the best way to get them because it contains strict version, without any conflicts (which is not the case if you take requirments in each subproject). But it only addresses runtime dependencies.

[FRidh]

I use pypi2nix to generate nix python packages skeletons.

@nlewo how about using pypi2nix for this entirely? How do you use these packages? Do you import the Python modules, or are they more like applications? And what about keeping it out of Nixpkgs but use it e.g. as an overlay?

A lot of dependencies needs to be updated. This will temporary
breaks other openstack components (Neutron, Nova,...). Can we
just mark them as broken?

AFAIK they were broken already

I needed to add specific version of already existing packages. In
particular, the version 2.12.5 of pythonPackages.requests. This is
required by oslo.policy: oslo.policy requires
requests!=2.12.2,!=2.13.0,>=2.10.0. But to avoid conflict, I also
add to use this modules version as a dependency of some other
packages, such as sphinx. I then introduce a new version of
sphinx. I added the suffix _openstack-ocata to all of this kind of
packages. Is it ok to add these kind of dependencies?

Let's not do this in python-packages.nix. We've been trying real hard to have just a single version of each package. Again, I think it may be better to use an overlay instead.

What about python version? At this moment, I only used python
2.7. I don't plan to run Keystone with python 3.x so I don't need
its dependencies to work with python3.x.
Could I disable them for python3?

If you're going to maintain these expressions, then that is your choice. But if you're not going to support 2 or 3, be explicit about it.

Python package hacking needs really strict dependencies. I had
to add them especially for this package.

How come they're so strict? Is it because the developers just use something like pip freeze, or is it because they depend on packages that changed API?

Several checkPhase are disabled because there is still a circular
import problem that I plan to solve.

if its documented 👍

[nlewo]

@FRidh I didn't want to use entirely pypi2nix mainly to share these packages with the community (feedback, security, tests infra,...). Moreover, there are also non python dependencies and some nixos modules that use Openstack applications.
All these python packages are mainly used by applications.

I don't understand why it is an objective to only have a single version of python package even if I understand it is an objective to minimize the number of versions. What is the problem of having several versions of the same modules? It seems a goal of nix is to be able to manage them at runtime. Why it is not the case at "build time"?

Regarding the package hacking, their doc is saying: "Hacking pins its dependencies, as a new release of some dependency can break hacking based gating jobs. This is because new versions of dependencies can introduce new rules, or make existing rules stricter."

I will have a look at overlays since I never used them. And I will also try to use https://github.com/garbas/nixpkgs-python.

[FRidh]

Nix can manage multiple versions of a package but Python cannot. Having multiple versions of a package in one set increases the risk of having multiple versions in a closure. This has already been happening too often.

[nlewo]

@FRidh I created a bunch of derivations with pypi2nix that are then used to build keystone. These derivation are not visible from the rest of the world. Basically, there is a directory keystone which contains two files: the requirements_generated.nix file is generated by pipy2nix and contains all deps of keystone. The default.nix file has been manually created, contains the keystone expression and imports the requirements_generated.nix file.
There are 88 packages in the requirements_generated.nix for keystone. Since pypi2nix doesn't handle test requirements, tests are disable. However, the keystone nixos test succeeded.
I need to rework a bit how the file is generated and how it is imported. I would like to have a first feedback to avoid to go in a wrong direction.
But, it works really well and it is really simple to package like this :)

Note: I also did it for openstackclient.
Note: a wip branch.

[FRidh]

How do you work with keystone? Do you import some of its modules? If so, then when you combine them with other packages you may end up with multiple versions of a package in a single closure.

[nlewo]

I don't manually import modules. I only use the keystone progam as here.

[FRidh]

Do you know if importing the modules is supported and if importing the modules is commonly done? If either is none, then we can package keystone as an application, and you can use pypi2nix for it in-tree without problems.

[nlewo]

I'm not sure to understand your question.
From the user and administrator of openstack point of view, it is not necessary to import module. He generally uses openstack applications. An openstack dev would like to manually load modules but this is not in the scope of this issue: I first would like to provide a simple openstack application stack.

[nlewo]

@FRidh I submitted #25898 which is the openstackclient generated with pypi2nix as a first example. This is also a application dependency of the keystone nixos test.

[nlewo]

Currently no time to work on this. And I will prefer to add it into nixpkgs-python repository.