Adding object store plugin for Rucio #17156

SergeyYakubov · 2023-12-08T18:28:22Z

Object store plugin for Rucio data management system (https://rucio.cern.ch).

It requires a Rucio cluster to be deployed and configured, which is not a trivial task. It works in our environment but might need adaptation to other environments/configurations. For tests, I created a standalone Rucio Docker image.

How to test the changes?

(Select all options that apply)

I've included appropriate automated tests.
This is a refactoring of components with existing test coverage.
Instructions for manual testing are as follows:
1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

test/integration/objectstore/_base.py

SergeyYakubov · 2023-12-08T18:55:09Z

Struggling with dependencies for rucio-clients. The recent versions (1.32+) require Python>=3.9, there is 1.29 LTS version that works with Python>3.6 but it requires requests>=2.20.0,<=2.27.1 (and probably something else). Poetry cannot resolve dependencies, pip install works for me (on Python 3.10, but also gives warnings about dependencies).

Any ideas on what can be done here?

bgruening · 2023-12-09T19:50:10Z

@SergeyYakubov cool stuff.

Do you think it is possible to build this dependency in theory for older Python version? We do build for a few exceptions our own wheels. https://github.com/galaxyproject/wheelforge

jmchilton · 2023-12-09T20:06:42Z

My alternative advice would just be to assert at runtime that this object store only works with Python 3.9+. If that is your target platform - I don't think supporting older Pythons is important for you to do. If someone else comes along and wants to use it on an older Python we can resolve that issue then - but I think it is totally fine for newer features to only support newer Pythons.

Bring in all the dependencies conditionally (ala https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/jobs/runners/util/pykube_util.py#L7) and then throw an exception at runtime if they are not available (ala https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/objectstore/pithos.py#L109).

SergeyYakubov · 2023-12-13T14:32:53Z

Ok, I think we should be good now.
CodeQL complains about passwords, also for other object stores - not sure what to do here.
Other fails do not seem to be related.

I do not see integration tests for Python 3.11, only unit tests, so my tests for Rucio object store are being skipped. Can we add integration tests for 3.11 (or anything above 3.8)

bgruening · 2024-02-14T10:03:14Z

@SergeyYakubov can you please rebase your PR. I hope we can than see a few more tests to pass.

mvdbeek · 2024-02-14T12:26:13Z

lib/galaxy/dependencies/__init__.py

+        return sys.version_info >= (3, 19)
+
+
+#        return "rucio" in self.object_stores


Suggested change

return sys.version_info >= (3, 19)

# return "rucio" in self.object_stores

return sys.version_info >= (3, 9) and "rucio" in self.object_stores

?

Though maybe skip the version_info check, if the dependency is not available we get a better error

If we skip version_info check, the GitHub integration tests will be failing, unless there is a better way to disable them for Python <3.9

You can check what python version you're in setUpClass of BaseRucioObjectStoreIntegrationTestCase and raise unittest.SkipTest("rucio requires python >=3.9")

mvdbeek · 2024-02-14T12:28:48Z

test/integration/objectstore/_base.py

+        "code.ornl.gov:4567/ndip/public-docker/rucio:1.29.8",
+    ]
+    subprocess.check_call(rucio_start_args)
+    time.sleep(20)


Is there a better way to check that the instance is up ?

mvdbeek

Looks great, just some minor stuff to fix here.

https://rucio.cern.ch

SergeyYakubov · 2024-02-14T18:15:02Z

You can check what python version you're in setUpClass of BaseRucioObjectStoreIntegrationTestCase and raise unittest.SkipTest("rucio requires python >=3.9")

Oh, actually, that was not the problem (was quite some time ago, I already forgot :)). The problem was that we configure the Rucio object store during the tests, i.e. after Galaxy was started. So, "rucio" in self.object_stores is False and it does not install Rucio clients and the tests fail. We might want to unconditionally install them if the Python version is new enough.

The same problem should be with irods, but it is also present in dev dependencies, so it works.

SergeyYakubov · 2024-02-14T19:11:45Z

@mvdbeek - any ideas why rucio-clients cannot be installed during the tests? The error comes from here, but I can build and install rucio-clients for Python 3.12 on my machine and also in Docker on Ubuntu 22.04

bgruening · 2024-02-16T07:57:42Z

@SergeyYakubov I do not know why. But I would like to use the opportunity to point you to https://github.com/galaxyproject/wheelforge . With this repo, we can create wheels if they are missing in pypi and publish them at https://wheels.galaxyproject.org.

Maybe that helps?

SergeyYakubov · 2024-02-16T14:42:19Z

Ok, I know what the problem is. In GitHub workflows galaxy root (with space) folder is used as a base directory ( see e.g. here.

And Rucio does not expect that during setup, so we have errors.

Was there a reason for having a folder with spaces? Maybe rename it for the sake of the future generations of developers' mental health :). Although this is a valid name, and with that, we test whether it works.

I can also disable the tests (add "rucio" in self.object_stores as @mvdbeek suggested earlier) for now (there are no integration tests with Python 3.12 anyway) and create an issue in Rucio's repo.

nsoranzo · 2024-02-16T14:48:51Z

Was there a reason for having a folder with spaces?

To find this kind of issues before they reach the end users/admins :)

SergeyYakubov · 2024-02-16T15:31:24Z

created an issue rucio/rucio#6499

mvdbeek · 2024-02-16T16:17:52Z

In the meantime building a wheel like @bgruening suggested would sidestep this: https://gist.github.com/mvdbeek/91a9ab31b19652dc1f54075d3b185d21

mvdbeek · 2024-02-16T16:19:45Z

galaxyproject/wheelforge#32

mvdbeek · 2024-02-16T16:22:03Z

hmm, guess it makes sense that we don't need to build pure-python wheels ... it might be the easiest upstream fix though.

SergeyYakubov · 2024-02-16T17:08:04Z

hmm, guess it makes sense that we don't need to build pure-python wheels ... it might be the easiest upstream fix though.

yeah, I created a PR in the Rucio repo. Hope it'll be approved soon

SergeyYakubov · 2024-02-19T16:33:07Z

Ok, the only failing test is now CodeQL. The problem is in "password"/"secret" keywords. Not only in Rucio, but also in S3 and iRods. I don't think we can change them now - will break all deployments that use those stores. Or can we? Alternatively, is there a way to make an exception for CodeQL?

@bgruening - what are your thoughts on it?

github-actions · 2024-03-05T16:07:59Z

This PR was merged without a "kind/" label, please correct.

bgruening · 2024-03-05T16:40:57Z

Congratulations @SergeyYakubov!

With the merge of galaxyproject#17156 , `rucio-clients` was listed as a conditional requirement, but it was actually installed almost everywhere (the only condition was `sys.version_info >= (3, 9)`), independently of its use for an object store. This also installs `rucio-clients` as a dev dependency, otherwise the `test/integration/objectstore/test_rucio_objectstore.py` integration tests would fail. Pinned dependencies are updated in the next commit. Fix galaxyproject#17695 .

github-actions bot added area/testing area/objectstore area/testing/integration labels Dec 8, 2023

github-actions bot added this to the 23.2 milestone Dec 8, 2023

github-advanced-security bot found potential problems Dec 8, 2023

View reviewed changes

test/integration/objectstore/_base.py Fixed Show fixed Hide fixed

mvdbeek modified the milestones: 23.2, 24.0 Dec 19, 2023

mvdbeek reviewed Feb 14, 2024

View reviewed changes

mvdbeek approved these changes Feb 14, 2024

View reviewed changes

SergeyYakubov added 5 commits February 14, 2024 09:03

Adding object store plugin for Rucio

0beef65

https://rucio.cern.ch

make rucio conditional dependency

0b9d28e

install Rucio for python>3.9,catch rucio import in extra file

2d8e731

rename password variable

f23ab02

fix lint

3dd7a70

SergeyYakubov force-pushed the rucio-plugin branch from a4127ca to 3dd7a70 Compare February 14, 2024 14:03

SergeyYakubov added 2 commits February 14, 2024 13:24

refactor tests skips, better wait to rucio to start

442632f

update rucio-clients to 33.5.0

e43b399

bump rucio-clients version to 33.6.0

332ba86

jdavcs added the area/backend label Mar 1, 2024

mvdbeek merged commit abe930a into galaxyproject:dev Mar 5, 2024
53 of 54 checks passed

nsoranzo added highlight/admin Included in admin/dev release notes kind/enhancement labels Mar 5, 2024

nsoranzo mentioned this pull request Mar 13, 2024

[RFC] rucio-clients dependency issues #17695

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding object store plugin for Rucio #17156

Adding object store plugin for Rucio #17156

SergeyYakubov commented Dec 8, 2023

SergeyYakubov commented Dec 8, 2023

bgruening commented Dec 9, 2023

jmchilton commented Dec 9, 2023

SergeyYakubov commented Dec 13, 2023 •

edited

bgruening commented Feb 14, 2024

mvdbeek Feb 14, 2024

mvdbeek Feb 14, 2024

SergeyYakubov Feb 14, 2024

mvdbeek Feb 14, 2024

mvdbeek Feb 14, 2024

mvdbeek left a comment

SergeyYakubov commented Feb 14, 2024

SergeyYakubov commented Feb 14, 2024

bgruening commented Feb 16, 2024

SergeyYakubov commented Feb 16, 2024 •

edited

nsoranzo commented Feb 16, 2024

SergeyYakubov commented Feb 16, 2024

mvdbeek commented Feb 16, 2024

mvdbeek commented Feb 16, 2024

mvdbeek commented Feb 16, 2024

SergeyYakubov commented Feb 16, 2024

SergeyYakubov commented Feb 19, 2024 •

edited

github-actions bot commented Mar 5, 2024

bgruening commented Mar 5, 2024

		return sys.version_info >= (3, 19)


		# return "rucio" in self.object_stores

Adding object store plugin for Rucio #17156

Adding object store plugin for Rucio #17156

Conversation

SergeyYakubov commented Dec 8, 2023

How to test the changes?

License

SergeyYakubov commented Dec 8, 2023

bgruening commented Dec 9, 2023

jmchilton commented Dec 9, 2023

SergeyYakubov commented Dec 13, 2023 • edited

bgruening commented Feb 14, 2024

mvdbeek Feb 14, 2024

Choose a reason for hiding this comment

mvdbeek Feb 14, 2024

Choose a reason for hiding this comment

SergeyYakubov Feb 14, 2024

Choose a reason for hiding this comment

mvdbeek Feb 14, 2024

Choose a reason for hiding this comment

mvdbeek Feb 14, 2024

Choose a reason for hiding this comment

mvdbeek left a comment

Choose a reason for hiding this comment

SergeyYakubov commented Feb 14, 2024

SergeyYakubov commented Feb 14, 2024

bgruening commented Feb 16, 2024

SergeyYakubov commented Feb 16, 2024 • edited

nsoranzo commented Feb 16, 2024

SergeyYakubov commented Feb 16, 2024

mvdbeek commented Feb 16, 2024

mvdbeek commented Feb 16, 2024

mvdbeek commented Feb 16, 2024

SergeyYakubov commented Feb 16, 2024

SergeyYakubov commented Feb 19, 2024 • edited

github-actions bot commented Mar 5, 2024

bgruening commented Mar 5, 2024

SergeyYakubov commented Dec 13, 2023 •

edited

SergeyYakubov commented Feb 16, 2024 •

edited

SergeyYakubov commented Feb 19, 2024 •

edited