Building python3 based packages for RHEL7 fails

[GuusHoutzager]

Hi,
Looks like something is going wrong when trying to build RPMS with python 3 on RHEL7.
I'm running 'python3 setup.py bdest_rpm'. Stuff seems to be going well, but it goes wrong when handling the man pages.
It copies them to the right directory, but then it runs /usr/lib/rpm/redhat/brp-compress which compresses them. During the next step it fails, because the files are now called for example barman.1.gz instead of barman.1. So then I get a bunch of error messages like this:
error: File not found: /home/vagrant/barman-2.12/build/bdist.linux-x86_64/rpm/BUILDROOT/barman-2.12-1.x86_64/usr/share/man/man1/barman.1
(and the same for the other man pages)
And then it stops:
error: command 'rpmbuild' failed with exit status 1
So no rpms :(
Is there a way I can fix this? Or is this a bug in the package? Thanks!

Cheers,
Guus

[amenonsen]

I can reproduce this failure in a centos:7 Docker container after installing python3 and rpmdevtools, but python3 setup.py bdist_rpm succeeds on my Debian machine. I don't know what the problem is, so I can't immediately help. I'll leave this open for a while in case anyone else has any ideas.

Can you not use the Barman packages from PGDG instead?

[GuusHoutzager]

Thanks for your response! Which packages do you mean? I usually download what I think are the official ones via https://www.pgbarman.org which hosts them on sourceforge. But they only provide python2 packages for RHEL7.
I'm running into some issues and I want to make sure it's not a python2 thing, hence the attempt to create python3 packages for rhel7. The rhel8 packages don't work out of the box on rhel7, tried that.

[amenonsen]

@GuusHoutzager I was suggesting to use the RPM packages from dl.enterprisedb.com or yum.postgresql.org.

Anyway, I tried this again with the Barman 2.13 release using Python3 on Centos7, and the problem is still there with python3 setup.py bdist_rpm:

+ /usr/lib/rpm/check-buildroot
+ /usr/lib/rpm/redhat/brp-compress
+ /usr/lib/rpm/redhat/brp-strip-static-archive /usr/bin/strip
+ /usr/lib/rpm/brp-python-bytecompile python3 1
Bytecompiling .py files below /root/barman-2.13/build/bdist.linux-x86_64/rpm/BUILDROOT/barman-2.13-1.x86_64/usr/lib/python3.6 using /usr/bin/python3.6
+ /usr/lib/rpm/redhat/brp-python-hardlink
+ /usr/lib/rpm/redhat/brp-java-repack-jars
Processing files: barman-2.13-1.noarch
error: File not found: /root/barman-2.13/build/bdist.linux-x86_64/rpm/BUILDROOT/barman-2.13-1.x86_64/usr/share/man/man1/barman.1
error: File not found: /root/barman-2.13/build/bdist.linux-x86_64/rpm/BUILDROOT/barman-2.13-1.x86_64/usr/share/man/man1/barman-cloud-backup.1
error: File not found: /root/barman-2.13/build/bdist.linux-x86_64/rpm/BUILDROOT/barman-2.13-1.x86_64/usr/share/man/man1/barman-cloud-backup-list.1
error: File not found: /root/barman-2.13/build/bdist.linux-x86_64/rpm/BUILDROOT/barman-2.13-1.x86_64/usr/share/man/man1/barman-cloud-restore.1
error: File not found: /root/barman-2.13/build/bdist.linux-x86_64/rpm/BUILDROOT/barman-2.13-1.x86_64/usr/share/man/man1/barman-cloud-wal-archive.1
error: File not found: /root/barman-2.13/build/bdist.linux-x86_64/rpm/BUILDROOT/barman-2.13-1.x86_64/usr/share/man/man1/barman-cloud-wal-restore.1
error: File not found: /root/barman-2.13/build/bdist.linux-x86_64/rpm/BUILDROOT/barman-2.13-1.x86_64/usr/share/man/man1/barman-wal-archive.1
error: File not found: /root/barman-2.13/build/bdist.linux-x86_64/rpm/BUILDROOT/barman-2.13-1.x86_64/usr/share/man/man1/barman-wal-restore.1
error: File not found: /root/barman-2.13/build/bdist.linux-x86_64/rpm/BUILDROOT/barman-2.13-1.x86_64/usr/share/man/man5/barman.5

I investigated a little more and found this setuptools issue that looks relevant. Indeed, the file build/bdist.linux-x86_64/rpm/BUILDROOT/barman-2.13-1.x86_64/usr/share/man/man1/barman.1.gz does exist in my container.

However, given the existence of easily-available RPMs of Barman from multiple sources, and the fact there doesn't seem to be a single authoritative workaround/fix for the problem upstream (but just a collection of varying hacks), this is not exactly a high priority. But I'm leaving the issue open anyway.

[GuusHoutzager]

Hi,
Thanks for looking at this again. Yes, there are available packages from multiple sources, but all of them for RHEL7 are for python 2. We're having a few issues with Barman and we're not sure if that's something inherent to Barman, or if it has something to do with python 2. Hence the attempt to run it with python 3. We should probably try with rhel8, because the barman packages for that are built with python 3. When I logged this issue earlier this year, we didn't have rhel8 available yet in our landscape, but we do now.
Cheers,
Guus

[amenonsen]

OK, trying it on RHEL8 makes sense, let us know how it goes. Also:

We're having a few issues with Barman and we're not sure if that's something inherent to Barman, or if it has something to do with python 2.

What sort of problems did you encounter?

[GuusHoutzager]

One of the issues we're having is issue #308 which I saw you're going to look at. The other one I haven't logged a ticket for but it's kinda related. If you run the default barman cron job that runs every minute but the previous run hasn't completed yet, these runs "pile up" somewhere inside barman, so when the run completes, it runs all of the "missed" runs in very quick succession (within a few seconds). This means barman logs in to the database potentially a lot of times in quick succession, making it cross some (default) limits which in turns generates alarms for our postgres databases. I wrote a wrapper script around the "barman cron" cron.d entry that uses locking, so it doesn't kick it off if the previous run hasn't completed. But I don't see the need for these runs to "pile up" inside barman, so hopefully that can be fixed too.

[amenonsen]

If you run the default barman cron job that runs every minute but the previous run hasn't completed yet, these runs "pile up" somewhere inside barman, so when the run completes, it runs all of the "missed" runs in very quick succession (within a few seconds).

Each invocation of barman cron tries to acquire a lock and goes to sleep because there's an earlier instance running, which explains the "pile up". I'll look into this code a bit more, but given that we can expect barman cron to be reinvoked very often, I would think it's not unreasonable to either limit the number of waiters or just timeout and exit after some time.

This means barman logs in to the database potentially a lot of times in quick succession, making it cross some (default) limits which in turns generates alarms for our postgres databases.

…but this I'm a little confused by. Why/how is Barman logging in to your database when you run barman cron?

[amenonsen]

(Neither of these problems sounds like it is likely to be anything to do with Python 2 vs. 3.)

[GuusHoutzager]

Yeah good question. I'm not 100% sure why it does this, maybe to verify settings (replication slot maybe?)? I'm very sure that it does this, because I can correlate the logins in the database to what barman is doing when you put the logging in debug mode.

[amenonsen]

Right, OK. I'm going to close this issue because this problem affects only setup.py bdist_rpm with Python2 on RHEL7, there's a known disutils bug with no obvious resolution, nobody has suggested a way to fix it, and there are other ways available to install Barman (setup.py install in the worst case, or RPMs from other sources—the EDB repositories do contain Barman packages that work with Python 2 on RHEL7, or you could rebuild your own RPM using the PGDG spec file if you really wanted, or use fpm to make an RPM), and your actual problems aren't anything to do with Python 2 vs. 3 anyway (you're welcome to open separate issues about them if you want to include more diagnostic information in the report).

I did spend time trying (and failing) to find a solution. It would have been nice to fix, but this has reached the point where it feels like going a little too far to fix a Python 2 problem for an installation mode that we don't otherwise support to begin with.