Friendly system wise compilation needed

[dirteat]

Hi there
it is a Royal Mess to compile recent version of astropy under a unix system due to its obsessive bundling of every possible library and bugs all over the setup system (It used to be fine for all versions <= 4.0.1).

One of your contributor "Olebole" is an expert in this and it is only thanks to him that I succeeded a build for current 4.2 version. I have been using its patches, published on the Debian repository, more than 12 are required to just not getting a crash while running python setup.py build! At some point I was wondering if I was compiling IRAF-2.16 or python3-astropy ;)

They are there: http://deb.debian.org/debian/pool/main/a/astropy/astropy_4.2-1.debian.tar.xz

Please merge them, or ask him consultancy if you need to understand how shared/system libraries works.

Thank you.
Best,
Chris.

[mwcraig]

Thanks for the suggestion @dirteat. If you or @olebole have some concrete suggestions in a PR we could take a look at them. Generally I would say that our goal is to make installation easy but that we delegate out responsibility for making packages for specific OS. Debian is an example of that -- as you point out, they have some strict rules about not bundling binaries that available as separate packages. Some other parts of our packaging ecosystem (conda packages) are also moving in that direction.

One of the things that prevents us from completely unbundling, though, is the lack of support in pip for non-Python packages. Since pip is the canonical tool for installing Python packages I cannot imagine unbundling everything for the foreseeable future.

That said, it could perhaps be made easier in the build process to choose to build with system, instead of bundled, libraries. We also overhauled the build system in 4.1, so there may well be some things that were broken, or whose breakage became visible, in the transition.

A more specific bug report including what system you are running on, the error log you are hitting, etc, would be very helpful.

It may be that we have some pre-packaged binaries for that system that we could point you towards.

[dirteat]

Thanks for the answer. The reason why I am building it is for distributing it over a linux distribution. I don't need specific help on an error message in particular, but for instance, it is impossible to purely build it against system libraries in spite of setting all the relevant environment variables. Most of the patches of Olebole are fixing issues related to that, just have a look to them, you'll see what they fix ( I would need to open a few bug reports here to summarize, I have not the courage).

Although I have interest in astronomy, I am not using astropy intensively, so if you guys decide to go to pip/conda only, I suspect you're targeting professional only; then we will just stop distributing it. I've opened this feature request just for you to be aware of having a (nice) piece of software not compiling fine in a system environment means it will most likely be abandoned by distributions at some point.
It is certainly not the place here to discuss why pip blind downloading is not very clever, neither very respectful of people not having fast internet connexions. Not talking about the fact that X users on the same system will install X times the same packages etc...
Anyway, I am also distributing iraf-community edition, and right now, it is actually easier to compile. Quite an irony :)

[mhvk]

@dirteat - Thanks for the further explanation. As @mwcraig notes, we generally rely on different distributions to package, with help from @olebole and @sergiopasra for debian and redhat, resp. They both already helped to reduce dependencies, remove patches, etc., but it is not a fast process, nor super-easy, since we also have to be sure that we can install properly on Windows, MacOS, etc. PRs that make this process even a little bit better are definitely welcome.

Note that right now, for Debian like systems, we actually test that compilation using system libraries is possible in weekly cron, using the code below (we combine this with a test for non-amd64 architectures, but should work for all). In principle, we could document that, though I would worry that it would soon get out of date (but perhaps just pointing to the below code would help). Certainly, the idea for that is that it sets up a state where pip needs to pull in very little indeed. Again, suggestions/PRs very welcome.

astropy/.github/workflows/ci_cron_weekly.yml

Lines 98 to 118 in 9c79f08

	install: |
	echo "deb http://deb.debian.org/debian buster-backports main" >> /etc/apt/sources.list
	apt-get update -q -y
	apt-get install -q -y git \
	g++ \
	pkg-config \
	python3 \
	python3-configobj \
	python3-numpy \
	python3-ply \
	python3-venv \
	cython3 \
	wcslib-dev/buster-backports \
	libcfitsio-dev \
	liberfa1
	run: |
	python3 -m venv --system-site-packages tests
	source tests/bin/activate
	ASTROPY_USE_SYSTEM_ALL=1 pip3 install -e .[test]
	python3 -m pytest

[olebole]

I usually forward patches to Astropy, if I think that they are useful to be included in the upstream source. However, most patches are really specific for Debian, and mainly ensure that the tests succeed under our circumstances. Specifially:

• don't install showtable: Has a conflict with another executable that is called "showtable"; there is however a discussion in astropy to rename the script (Rename showtable? #7233)
• Disable remote access to get M3 coords: This would implicitly require remote access, which is not allowed during Debian build. This is Debian specific, so no need to apply this upstream. See also [3.1] Remote access when building documentation #8311
• Allow XFAIL tests to pass: Some tests fail on some Debian platforms, but succeed on others. So, we pragmatically allow xfail tests to pass.
• Don't turn deprecation warnings into errors: While from the POV ov developers, it is great to be strict here, as a distrubutor, I can (and sometimes must) live with deprecations: otherwise new versions of dependencies would cause the CI tests to fail, and the updates can't migrate. Debian specific --> not forwarded
• Make some votable tests xfail that fail under mips64el: This is basically the unresolved 4.0rc2: Deprecation warnings during VOTable tests on MIPS platform #9749
• Weaken leap second validity requirement for tests: This is another "real-world" problem, that we cannot really ensure here that the system is updated enough to keep that latest leap second update. Since this does not show an astropy bug, I decided to disable this check.
• Remove intersphinx mapping: This is a pragmatic patch. In Debian, we can't access external web pages, but I also don't want to have the documentation dependent on a number of other documentation packages. This could be improved however, if someone wants :-)
• Use system wide configobject+validate: In Debian, configobj and validate are available as packages, so we use them. This one actually could be proposed to be applied upstream.
• Use system ply instead of local copy: same
• Unconditionally skip wcsapi test if compilation fails: This test doesn't work in CI and is already disabled for Travis & Co. I just pragmatically disabled it unconditionally to remove the failure on Debian's CI.
• Skip test_timer (obsolete): See [4.1RC2] test_timer failed on s390x #10847 for discussion
• Ignore FPE warnings for armel build: See [4.2rc1] Test failures on armel: no FPEs #11011; this is rather a problem of the (IMO obsolete for astropy) armel Debian platform.

Summarized, I don't see the big problem here -- two patches may be discussed further, but the rest is really dealing with Debian peculiarities. If you (@dirteat) see this different, feel free to open an issue on Astropy, please put me in Cc in this case.
What is the distribution you working on, BTW?

Concerning IRAF: There I am my own upstream (I mantain both iraf-community and the Debian package, and even then I have a number of patches specific for the Debian package -- not really less than for Astropy. This is just a common problem of complex software to have it well-build on a platform.

[mhvk]

@olebole - thanks so much for the full list! It would be nice if we can make the use of the packages bundled in extern easier (or perhaps move to just relying on pip/conda to install it as needed); for the largest of the extern bundled packages, jquery, there is already an issue open - #10420.

[embray]

@olebole For the intersphinx mapping, I wonder if there might be some way to package the documentation for the other projects it links to (e.g. Python) and run the doc build with some kind of local web proxy which handles this. Probably not worth the effort though.

[olebole]

@embray, you are right; and often these are already packaged. However, this creates additional dependencies, and I am too lazy to maintain them. I must say that I am not sure how useful the local documentation is at all, as it is available online as well (and I personally just use the online docs here). And the -doc packages are installed rarely; for astropy f.e. we have >1400 package installations (Python 3), but only 36 installations reported.

But I would accept patches to do this properly ;-)

[embray]

Local documentation has its uses, though I agree it's rare :)

[dirteat]

Hi there,
thank you for the follow-up. I had to use these ones to have a successful compilation against system libs:

Use-system-ply-instead-of-local-copy.patch
Use-system-wide-configobject-validate.patch
python-astropy-system-configobj.patch

Best,
Chris.

[olebole]

@dirteat the third one does not exist for astropy_4.2-1 in Debian.
For the others two patches: Astropy should work without them. If not, it is a bug in Astropy. Can you give a proper description what fails without these patches? Which sources did you use (pypi? github?)?