flux python packaging rollup

[trws]

there are a few related things floating around, basically all of which could be solved with some packaging work

• python: add a pip requirements file with our python dependencies #2841
• Add Python bindings to PyPI for easy user-installation via pip #2145
• Python: flux bindings with multiple Python versions #4254
• Add Python black formatter into docker images #2657

the current situation is, to my shame, my fault

well, at least the python part, I tried really, really hard to convince Jim and Mark to let me rebuild the build system, in anything other than autotools, and didn't manage it, so we still have to deal with that at least some

This is where the python bindings live: https://github.com/flux-framework/flux-core/tree/master/src/bindings/python

you'll notice that there is a setup.py in there, but it's kinda vestigial, it only works inside the autotools build right now, which is what we really need to fix: https://github.com/flux-framework/flux-core/blob/master/src/bindings/python/setup.py

this is the main reason why: https://github.com/flux-framework/flux-core/blob/master/src/bindings/python/_flux/Makefile.am

it uses this script

to generate a header that cffi can consume without issues to generate a 100% complete binding from python to all flux interfaces, without the C-only folk having to touch python to add them

this is... not normal, it's also probably both over and under engineered at the same time, but it was a political coup for me to get that in there, because it means that python developers can do absolutely anything with flux without writing C (sometimes with difficulty, but still) and the bindings are always complete, and always stay at least generating because core can't build without them building

the downside to this is that the python build must have access to flux headers, they don't have to be in the source tree, could be installed, but basically python-flux-core would need flux-core-dev if we're talking about ubuntu packages

the best reference for managing this stuff in python packages is probably the cffi documentation (cffi is awesome, makes ctypes look useless in comparison) https://cffi.readthedocs.io/en/latest/cdef.html

We use out-of-line, "API mode" currently, though we could do otherwise if you have ideas on making sure the right headers are available

the setup to build the cffi-compiled modules are all in _build.py files in the directory above, like this one for core: https://github.com/flux-framework/flux-core/blob/master/src/bindings/python/_flux/_core_build.py

sorry for the book btw... this is something I've wanted done for years and never had the time to finish, I'm actually really excited about it getting fixed

my thought now, since I've had a lot more time to think about it, is that we can go for a flit, poetry or hatch-based setup (I like poetry personally, but hatch is the new hotness and some like flit better for libs, if you have a preference go for it) so that we have an easy way to get all the python deps and dev tools and also an easy way to get the binding generation into pypi. Pretty sure we'll have to do a source-based distribution because it has to build against a specific flux to build the right interfaces, but any way to build the bindings for multiple pythons, and incorporate it separately into pip/conda/whatever would be awesome and also solve a whole lot of customer issues for our workflow users.

[vsoch]

A TLDR from @trws:

goal make it possible to pip install flux-core

1. figure out python dependencies from spack package or just testing it, encode those in requirements in a pyproject.toml so poetry or similar can load our dependencies
2. set up whichever packaging tool so it works with the python bindings after they have been built in the autotools setup, just in tree
3. get the packaging tool running the proper script to generate its header stuff and compile the modules without the autotools build
4. switch the autotools build to build the python components with poetry or whatever packaging tool
   of those doc: spelling in flux-start(1) #1 is basically the requirements.txt thing, the other three are the bigger goal broken down.

[ofek]

Let me know if you need help with Hatch! (creator)

[vsoch]

Would love that! We have an early PR here: #4576 where I got most of it working with setuptools, and that could install from pip via the .tar.gz produced! I'm actually not super familiar with hatch but I'd love to see how it works here!

[grondo]

Just bumping this issue because on our systems the default Python is not the system Python, so import flux does not work outside of a Flux instance. Since there are multiple/many Python versions and even multiple of the same version installed and used by users, allowing users to pip install Flux bindings may help.

[vsoch]

For the PR here, I did a few (rebasing, etc) and it was closed due to no activity: #4835.

And the setup (that I like a little bit better) I separated it entirely. I posted a full proposal with ideas in the slack a bit back if anyone wants to dig that up! https://github.com/vsoch/flux-python. I would say it makes sense to do a bunch of automated builds in the separate repository for each version of flux, and then have the releases coincide. The Python bindings in Flux don't technically need to be removed, but they can be built / distributed elsewhere I think.

[grondo]

I'm not knowledgeable (at all) about python packaging so I have few opinions here! I would need @trws opinion (and of course yours @vsoch). I was just noting that we're starting to get real requests for something like this.

[vsoch]

I think there are several ways to roll this - and the most important detail is that someone that needs to install a particular Python binding knows what version to use (to match with the flux-core code). I think my suggestion from a few months back was to try and build / distribute from a separate repository, and always build the bindings to coincide exactly with flux-core releases. We could try that sort of manually to start, and then eventually have something akin to what we have with the flux spack repository - a nightly run that checks for a new release, and if there is one, builds and releases the python package.

[grondo]

That sounds great to me!

[vsoch]

okay so here is what I probably need to get this underway:

1. the flux-python repository should probably be under the flux-framework org and not my personal account. I can try doing a transfer but I'm not sure that works if you can't make repos (I can't make repos here)
2. I'll need to be added as a manager to whomever owns the pypi identifier (this is me https://pypi.org/user/vsochat/)
3. then I'll test automation that can detect a new release
4. and all the pypi release automation onto that

We will want to have a loop that (likely I'll run locally) that will go back and build all the previous releases (up to some point).
And another idea - instead of steps 3 and 4, we could also start with a "test release" where sometimes shares a version of flux they have on a system, we build one package for it, and then test with it (to make sure it works). I think any of the approaches above will work, but the basics of what I need are pypi and the repository being rooted here I think.

And to explain the current repository, right now I put the flux core code directly in it - that's the "flux" folder at the top. The "src" folder are the headers from the PR that I had made to flux - they are modified to work with the setup.py there. For the automation likely we will want that code to be retrieved from the release and updated there. The one hard part to figure out is how to handle the flux root, here https://github.com/vsoch/flux-python/blob/570fc45773008949399f676d65d2e648acc39aca/setup.py#L335-L355. Maybe @trws has ideas for this, and I guess worst case (if it's consistent) we could build for specific systems, and actually just namespace the packages to them (e.g., flux-coral) or similar.

[grondo]

Ok I have created a flux-python repo with core team as maintainer (you are on that team I believe). I don't know who owns the pypi identifier, hopefully someone we know!

[vsoch]

Awesome! I’ll start doing the setup and testing soon.

[vsoch]

I think we are done with all of the linked issues here - I posted on the last two to follow up. We have a requirements file under scripts (first) and install black in the developer container environment with pre-commit (fourth) and the other two are done.

[vsoch]

I think we've addressed all the linked issues here - I'm going to close! @trws if there is something new / different to work on, let's open a new issue (and please ping me any time to chat about Python stuff)!