-
Notifications
You must be signed in to change notification settings - Fork 309
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for object storages in Auditor #2263
Comments
I've been looking into this a bit. While it is possible to list a (pseudo-)directory on all of the objectstore platforms supported by Rucio, this has disadvantages. It can't be done using signed URLs and HTTP access so the auditor would require special code for each objectstore type, would depend on their client libraries, and would need access to the objectstore credentials. It may also have a performance cost on some platforms since these aren't "real" directories but simply a filtered version of the flat object namespace. I think my preference would be to try the last n days using the pattern, as Dimitrios suggested. This avoids the issue I mentioned above, and also avoids adding extra complexity for site admins by allowing them to place the dump files in the same way as for other types of storage. Alternatively, using a static location would not be too bad - currently Rucio only ever downloads the latest dump anyway, so we don't lose much by only storing one. Does anyone have strong feelings about this? If not I can start implementing this. |
Motivation
Currently, the Auditor expects to be able to list directory entries in order to identify the latest dump. This is not possible on object storages.
Modification
Investigate and implement an alternative, such as:
The text was updated successfully, but these errors were encountered: