Bugfix release 5.4.1

[simonmichal]

Dear XRootD community,

I'm planning to cut the 5.4.1 with backports of bugfixes from the master branch, I'm creating this issue so we have a place where we can discuss the upcoming release.

The stable branch has been already updated and tagged:
https://github.com/xrootd/xrootd/tree/stable-5.0.x
https://github.com/xrootd/xrootd/releases/tag/v5.4.1-rc1

Here's the full list of new commits in the upcoming release (so far):
990fe19 [Docs] Update release notes for 5.4.1.
b339451 [CI] Add Python 2 bindings test job to GHA wokrflow
679c571 [CI] Add sdist building with publishing to CentOS 7 RPM stage
fd609b4 [PyPI] Try to use build to make sdist, else fall back to setup.py sdist
ef56fdc [Python] Try to install sdist with pip during packaging, fall back to setup.py
d9703f6 [Python] Remove unused Python imports
2a11053 [Python] Use CPython libraries over distuils
635c2bc [RPM] Add python3-setuptools to BuildRequires
8efb408 [CI] Update pip release script.
be0e03a Fix warnings about allocations exceeding maximum object size
cfef402 [Server] Prevent SEGV on busy systems due to missing lock call for background jobs.
052eb04 put /path/filenname in LFH and CDFH records (reverse the change)
744c7da [XrdCrypto] Generate DH parameters on first call to openssl3/XrdCryptosslCipher
b517fb7 [XrdCrypto] DH parameter generation cleanup
20f3dd1 [XrdCl] Add missing timeout argument to declarative operations.
e47077b Fix compiler warnings about pointers used after free:
8678fa2 Fix comments
5667770 Add support for static openssl libraries
ae83cdd [XrdCrypto] Generate DH parameters on first call to XrdCryptosslCipher
242fb7a [XrdCl] Make sure HS wait is not handled after channel has been TTLed.
857ecd0 Do not give up ownership of ChunkInfo
c18a937 [XrdCl] Add declarative PgWrite.
0bb72ff Fix warning about comparing arrays
8441abe Fix warning about dereferncing null pointer
02aea53 Fix warning about uninitialized variable
2ced499 simplify coding
f47afe1 change xattr name xrdec.chunkver to xrdec.strpver
b19d8f7 [CI] Remove Travis CI config file
d92759d [CI] Add GitHub Actions based CI
515566a combine two xattr update to one
cd6b2b0 [CMake] Add XrdSysTrace.hh to private headers.
8774b6d Update to XrdCl and XrdEC
8d4caf6 [XrdEc] Ensure parallel execution of Reader::Read is thread-safe.
f993b95 [IOEvents] Improve tracing.
45ee156 [XrdSciTokens] Modifying std::map invalidates iterator
0c9a3ca Fix compilation on GCC 4.8
830ec57 Add missing include
cc62abe Fix compilation when char is unsigned (armv7hl, aarch64, ppc64le, s390x)

Please let me know your thoughts!

Cheers,
Michal

[matthewfeickert]

Thanks @simonmichal for the nice summary. 👍 I don't want to hold up getting any releases out (code shipped is code used!) but if it is possible to get PR #1605 into v5.4.1 that would help things out in general for Python 3 related things, and with that in PR #1586 would just need to be rebased and cleaned up and would be ready for review.

[simonmichal]

@matthewfeickert : absolutely, I will have a look at #1605 first thing tomorrow morning, I think we absolutely still have time to accommodate both #1605 and #1586.

[simonmichal]

@matthewfeickert : I just merged #1605, let me know when #1586 is ready for review.

[simonmichal]

Following commits have been added to the stable branch:

5551cac [XrdCl] xrdfs: support rm of multiple files.
0fb70b1 [pip] Sanitize version to be PEP 440 compliant
0e8bc8c [Python] Use context manager for opening files
2cc617c [CI] Move xrootd.egg-info off PYTHONPATH
31e3d71 [Python] Install Python bindings with pip if available
a6dbe2b [CMake] Add PYTHON_VERBOSE flag to enable --verbose pip install
c6d0310 [CI] Enable PIP_VERBOSE CMake option for builds in CI
d004070 [RPM] Add python2-pip to BuildRequires
a6a1fda [RPM] Fix python packaging.
99a4b85 [CI] Add Fedora 35 rpm build to GitHub Actions CI workflow
f865d44 [CMake] Remove use of --record in Python installs
1fa31fd [Posix] Make sure pointer is set to 0 to avoid memory corruption.
3a73fec [Debian] Add python3-pip, python3-setuptools as required packages
686a5a6 [CI] Add CI for ubuntu-latest virtual environment
c64a92a Fix OSX build.
0360155 [XrdCl] xrdfs rm: fix log msgs, print status per file.

[simonmichal]

Following commits have been added to the stable branch:

0bac4b8 [CMake] Add XrdOuc/XrdOucPgrwUtils.hh to private headers.

[olifre]

I can't seem to install xrootd-scitokens from the current RC2 on Rocky 8:

Execution of '/usr/bin/dnf -d 0 -e 1 -y install xrootd-scitokens' returned 1: Error: 
 Problem: package xrootd-scitokens-1:5.4.1-0.rc2.el8.x86_64 requires xrootd-server = 1:5.4.1-0.rc2.el8, but none of the providers can be installed
  - package xrootd-server-1:5.4.1-0.rc2.el8.x86_64 requires xrootd-server-libs = 1:5.4.1-0.rc2.el8, but none of the providers can be installed
  - cannot install the best candidate for the job
  - nothing provides libjson-c.so.4(JSONC_0.14)(64bit) needed by xrootd-server-libs-1:5.4.1-0.rc2.el8.x86_64

Checking repos, I find:

# dnf info json-c
Last metadata expiration check: 1:00:49 ago on Sat Feb 19 19:54:43 2022.
Installed Packages
Name         : json-c
Version      : 0.13.1
Release      : 2.el8
Architecture : x86_64
Size         : 68 k
Source       : json-c-0.13.1-2.el8.src.rpm
Repository   : baseos
...

i.e. there is no json-c in version 0.14.

(using packages from https://xrootd.slac.stanford.edu/binaries/testing/slc/8/x86_64/ )

[simonmichal]

@olifre : thanks for reporting this problem, I tried on Centos Stream 8 and the install does work:

# dnf list xrootd-scitokens json-c
Last metadata expiration check: 0:09:08 ago on Mon 21 Feb 2022 05:40:54 PM CET.
Installed Packages
json-c.x86_64                                                                                                    0.13.1-3.el8                                                                                                @System        
xrootd-scitokens.x86_64                                                                                          1:5.4.1-0.rc2.el8                                                                                           @xrootd-testing

Let me tried it on Rocky.

[simonmichal]

OK, I can reproduce it on Rocky but have no idea why it wants json-c 14 on Rocky but is ok with json-c 13 on lentos stream 8 :|

[olifre]

Thanks! The only obvious difference to me is:

OS	version
CentOS 8 Stream	json-c-0.13.1-3.el8.src.rpm
Rocky 8	json-c-0.13.1-2.el8.src.rpm

[olifre]

@simonmichal Here's the answer:

Rocky 8

# rpm -q --provides json-c
json-c = 0.13.1-2.el8
json-c(x86-64) = 0.13.1-2.el8
libjson-c.so.4()(64bit)

CentOS 8 Stream

# rpm -q --provides json-c
json-c = 0.13.1-3.el8
json-c(x86-64) = 0.13.1-3.el8
libjson-c.so.4()(64bit)
libjson-c.so.4(JSONC_0.14)(64bit)
libjson-c.so.4(JSONC_0.15)(64bit)

[olifre]

@simonmichal And here's the cause:
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2001063 asked to introduce symbol versions, and this was done in:
https://git.centos.org/rpms/json-c/c/aa505d489ccc4ad2e2abfcc61b08b8f8b272c4f4?branch=c8s
That seems like a rather sensible change, but of course it's breaking backwards compatibility.

I think the summary is that packages built on CentOS X Stream may not work / not install on the latest release versions of any of the various rebuild distros. I'm also unsure about the "guarantees" of the Stream development workflow concerning backwards compatibility (after all, it's a testing ground before a point release, so it should be possible for breaking changes to be reverted).

On the other hand, I would expect it to be significantly more likely that packages built on one of the rebuild distros run fine on Stream.

[simonmichal]

@olifre : thanks for investigating! I tried rebuilding and installing the RPMs on rocky and it works like a charm:

# dnf list installed xrootd-scitokens json-c
Installed Packages
json-c.x86_64                                                                                                     0.13.1-2.el8                                                                                                 @System      
xrootd-scitokens.x86_64                                                                                           1:5.4.1-0.rc2.el8                                                                                            @@commandline

I will split the slc8 repo into stream8 and rhel8 and build the RPMs separately for both.

[olifre]

I will split the slc8 repo into stream8 and rhel8 and build the RPMs separately for both.

Perfect, many thanks! Of course that's more effort in the long run, but it seems to be the safest solution to build for both flavours separately.

[simonmichal]

Sorry for the delay, so I have add a slc8strm repo (this name has been chosen for convenience as it was easy to add it to current build scripts):
https://xrootd.slac.stanford.edu/dload.html
https://xrootd.slac.stanford.edu/binaries/xrootd-testing-slc8.repo
https://xrootd.slac.stanford.edu/binaries/xrootd-stable-slc8strm.repo

I tagged RC3 only to test the new rhel8 and stream8 builds, since everything seems now to be in order I will go ahead with the final release.

[simonmichal]

Dear All,

The XRootD 5.4.1 bugfix release has been tagged. Please follow the link below for more details:
http://xrootd.org/2022/02/23/announcement_5_4_1.html

Kind regards,
Michal