pip/pypi as an alternative to Conda

[CompRhys]

Would you be open to making GSAS-II pip installable. Our team is happy to work on the actions and minimal code changes needed to facilitate this. We do not want to use Conda in systems but do want to explore the use of GSAS-II to power some XRD workflows.

[briantoby]

@CompRhys in a word, yes. At one point @tacaswell was interested in this and a lot of reorg work was done to make the GSAS-II code more conventionally organized, but I think he wants to stick with conda now.

I know very little about how pip does installation, but I assume you would want to have pip do a local compilation rather than download binaries that can only match a small number of Python/numpy versions. In case it is of help, this script does a download and install: https://github.com/AdvancedPhotonSource/GSAS-II-buildtools/blob/main/install/gitcompile.py

Would you want help in creating a list of required packages?

[CompRhys]

I think the typical approach is that you just build wheels for all the different systems you want to support. If we go down this route we would likely adopt a similar CI scheme to sklearn where you can see their wheel builds here: https://github.com/scikit-learn/scikit-learn/blob/main/.github/workflows/wheels.yml

[tacaswell]

GSAS-II is currently able to be built as a normal Python package (the config in the pixi directory can build wheels). If you are willing and able to build locally it should "just work" today.

However, there are some technical issues with wheels (see https://pypackaging-native.github.io) and I am not sure that it is worth the effort to get cibuildwheel to work (given the fortran, scipy is a better example to look at the skimage).

conda-forge/staged-recipes#29742 is in-progress work to get gsas-ii packaged on CF. There is some issue getting fortran setup correctly in windows, but I'm more worried about getting the licenses sorted.

[CompRhys]

Maybe I am being daft but when I run pixi run install inside a fresh venv and make the compiler changes in the config file to use the right fortran compiler I have from homebrew on this mac and then try to run the tests in the venv they most all fail.

[briantoby]

I have not seen that. I'm not sure I want to diagnose pixi issues myself, but the self-tests use pixi on all platforms.

[tacaswell]

pixi provides its own environments and compilers and you should not expect it to install in to an environment it does not control.

Sorry, I was typing very quickly before a meeting. What I meant was to look at what the pixi tasks are doing for the correct incantations to build and install the wheels. If you have a venv with the right dependencies and compilers available you should be able to replicate that into your environment.

[CompRhys]

The issue here was that the tests are a little brittle in so far as when I tried to run them through vscode they break but if I run from the command line then they pass.

[briantoby]

No doubt I'm guilty then. I don't know why VScode would change things, but pytest does do some strange things.

If you learn anything more on what is going wrong with the tests, I'd be willing to revisit them.

[briantoby]

I did see some strange results with self-tests on a quick RHEL installation, but I was not in a good place to follow up on that. I will have to go back to that when I get a chance, but if you have seen any more on self-test issues, please do let me know.

I am beginning to think that prebuilt binaries are not such a good idea on Linux. (They are great for Mac & Windows), but on Linux I am seeing a lot of glibc version conflicts. I will probably change my install instructions to recommend use of the "gitcompile.py" script that downloads via git and compiles with meson. How does one handle system library version reliance with pipy wheels?

[CompRhys]

Having gone deeper I think this is a fundamental limitation that is directly a result of using WxPython and I am not sure that there's a good solution to distribution via PyPI. Closing this issue as a result.

[Jecrevin]

Actually, there's no need for publishing wheels onto PyPI, pip itself can directly install packages from Git (see VCS support for more info). I tested for only scripting usage, using the following command (notice fortran compiler needed):

pip install git+https://github.com/AdvancedPhotonSource/GSAS-II.git

can perfectly install GSAS-II into my python virtual env, and the import statement from GSASII import GSASIIscriptable as G2sc works perfectly without any configuration.

for GUI usage, I haven't tested yet, but I think just specifying optional dependency group, i.e., pip install gsas-ii[gui] git+https://github.com/AdvancedPhotonSource/GSAS-II.git can also fetch wxpython and build it locally (note gcc and some additional build tools are needed), it should work.

[briantoby]

@Jecrevin I had no idea this was possible and creates a potentially very useful way to install GSAS-II, particularly for Linux. Thank you very much for this. I am planning to revisit installation methods and documentation fairly soon and will look into this further. I can see I will need to learn more about how to better configure the pyproject.toml contents.

Just to be clear, there are a few aspects of installing GSAS-II:

• The most significant is the compilation of the binaries (or installation from pre-compiled .tar files). Without these, significant sections of GSAS-II (scripting or GUI) will not run. Certainly the self-tests will not complete.

  • compilation requires gfortran, gcc, cython & meson

  You must have them already installed or I don't think the above will work. I'll have to do some testing.

• For scripting the Python configuration is pretty trivial, as only numpy and scipy are needed, though PyCifRW is hard to live without and may as well be considered as required. (This is discussed in the ReadTheDocs manual.)

• For running the GUI, a few more packages, wxPython, matplotlib and PyOpenGL, are also needed (also documented), but there are no additional compilation needs.

• There are also a number of recommended, but not required packages. This needs to be reviewed in pyproject.toml file and facilitated.