Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache information about the namespace #39

Merged
merged 5 commits into from
May 11, 2024

Conversation

bbockelm
Copy link
Contributor

This PR adds a cache which tracks information about the various namespace prefixes.

On cache miss, the prefix and the list of endpoints are put into the cache (caching for up to 15 minutes). This information is reused on subsequent opens of the object.

Additionally, to ease my local testing, this registers the osdf:// and pelican:// schemes with fsspec, allowing this to be a valid mini-test:

import fsspec
with fsspec.open("osdf:///ospool/uc-shared/public/OSG-Staff/validation/test.txt") as of:
    print(of.read(1024))

The namespace cache keeps track of the endpoints associated with a
given prefix of the federation namespace.  It is a "ttlcache", meaning
the data will be periodically refreshed from the dirctor as it
becomes stale.

The cache can be updated to remove problematic endpoints; in the future,
this will allow us to intercept I/O failures and mark certain endpoints
as unusable.
@bbockelm bbockelm requested a review from turetske May 11, 2024 13:30
This hooks the cache manager into the exception handling of the
decorators, marking caches as bad when an exception is thrown.
@bbockelm
Copy link
Contributor Author

Just to get things going here, I'm going to merge.

We now have unit tests! 🎉

@bbockelm bbockelm merged commit 700b415 into PelicanPlatform:main May 11, 2024
1 check passed
This was linked to issues May 11, 2024
@bbockelm bbockelm mentioned this pull request May 11, 2024
@bbockelm bbockelm linked an issue May 11, 2024 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant