Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for object storages in Auditor #2263

Closed
dchristidis opened this issue Mar 1, 2019 · 1 comment · Fixed by #6362
Closed

Add support for object storages in Auditor #2263

dchristidis opened this issue Mar 1, 2019 · 1 comment · Fixed by #6362

Comments

@dchristidis
Copy link
Contributor

Motivation

Currently, the Auditor expects to be able to list directory entries in order to identify the latest dump. This is not possible on object storages.

Modification

Investigate and implement an alternative, such as:

  1. Try the last n days, based on the pattern
  2. Use another file as an index
  3. Use a static location and obtain the date from its metadata
@jamesp-epcc
Copy link
Contributor

I've been looking into this a bit. While it is possible to list a (pseudo-)directory on all of the objectstore platforms supported by Rucio, this has disadvantages. It can't be done using signed URLs and HTTP access so the auditor would require special code for each objectstore type, would depend on their client libraries, and would need access to the objectstore credentials. It may also have a performance cost on some platforms since these aren't "real" directories but simply a filtered version of the flat object namespace.

I think my preference would be to try the last n days using the pattern, as Dimitrios suggested. This avoids the issue I mentioned above, and also avoids adding extra complexity for site admins by allowing them to place the dump files in the same way as for other types of storage. Alternatively, using a static location would not be too bad - currently Rucio only ever downloads the latest dump anyway, so we don't lose much by only storing one. Does anyone have strong feelings about this? If not I can start implementing this.

jamesp-epcc added a commit to jamesp-epcc/rucio that referenced this issue Nov 3, 2023
jamesp-epcc added a commit to jamesp-epcc/rucio that referenced this issue Nov 3, 2023
jamesp-epcc added a commit to jamesp-epcc/rucio that referenced this issue Nov 3, 2023
jamesp-epcc added a commit to jamesp-epcc/rucio that referenced this issue Dec 11, 2023
jamesp-epcc added a commit to jamesp-epcc/rucio that referenced this issue Jan 29, 2024
jamesp-epcc added a commit to jamesp-epcc/rucio that referenced this issue Feb 2, 2024
jamesp-epcc added a commit to jamesp-epcc/rucio that referenced this issue Mar 29, 2024
jamesp-epcc added a commit to jamesp-epcc/rucio that referenced this issue Mar 29, 2024
jamesp-epcc added a commit to jamesp-epcc/rucio that referenced this issue Mar 29, 2024
@bari12 bari12 added this to the 34.2.0 milestone Apr 10, 2024
@bari12 bari12 linked a pull request Apr 10, 2024 that will close this issue
@bari12 bari12 closed this as completed Apr 10, 2024
voetberg pushed a commit to voetberg/rucio that referenced this issue Apr 15, 2024
labkode pushed a commit to labkode/rucio that referenced this issue Jun 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants