Remove upper bound version constraints for rucio-clients

[SergeyYakubov]

Description

Currently, Rucio clients have pretty strict requirements for third-party packages libraries. That makes it difficult for other applications to use Rucio client libraries.

So, it would be good to relax the upper-bound requirements in rucio-clients package

Motivation

A lot of dependencies are so strictly pinned that it prohibits upgrades of other packages and we might be missing out on critical security bugs. We might not even notice that because our automated updates (for which we use poetry) will only consider possible updates given all dependencies (Rucio ones and our software).

I think the rule of thumb is that when building an application, one should strictly pin dependencies; if building a reusable library, it's better not to pin your dependencies (or only set a lower bound / exclude known bad versions). This is discussed in detail in this blog.

Change

Remove upper bound limit for the dependencies needed to run rucio client

[bari12]

This is a complicated issue, and there probably is not one right answer for it.

While it is true that not pinning the dependencies might get some security issue quickly fixed, the recent supply chain attack(s) on the xz library showed also the other side of this coin, unpinned dependencies might ship compromised code into your workflows.

From a validation and test point of view the stricter dependencies also give us easy reproducibility in our CI and we can ensure that certain combinations of dependencies work. We did have less strict pinning before and ran into this problem more than once.

For that reasons, we will stay with this principle, and rather react to specific issues (issue with our pinned dependency version, or reported incompatibilities) when needed.

That being said, I also see the possibility to decouple the dependencies from the rucio clients from the rucio backend code (Right now, they both are based on the same requirements file) and be more relaxed for the clients. While this could still get you supply chain attacks, which could leak rucio authentication tokens, it is less of a critical issue on the client side, than a compromised server. Something we need to think about.

[mvdbeek]

unpinned dependencies might ship compromised code into your workflows.

The distinction here is that the client code can be used in application code, which itself pins dependencies strictly. I would really appreciate a way to install the client without strict dependency pinning.

For that reasons, we will stay with this principle, and rather react to specific issues (issue with our pinned dependency version, or reported incompatibilities) when needed.

This breaks automated updates and isn't an option for us as we'd have to check the reason for restricted dependencies manually every time we attempt to update our dependencies.

[bari12]

We had an internal discussion on it and will do the following:

• All Rucio backend products (Server, daemon, etc.) dependency management stays the same with pinned dependencies.
• For the clients, which only have a handful of dependencies, we will switch to a less strict dependency management to help in usecases like yours (usage as a library)

I don't have a timeline for this yet though.

[SergeyYakubov]

sounds great, thank you!