Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make caching optional #103

Closed
yarikoptic opened this issue Feb 16, 2024 · 0 comments · Fixed by #104
Closed

Make caching optional #103

yarikoptic opened this issue Feb 16, 2024 · 0 comments · Fixed by #104
Assignees

Comments

@yarikoptic
Copy link
Member

ATM local filesystem caching of accessed content is non-optional. To make fsspec work multi-threaded, heavy locking was introduced and we are using custom branch with that. That locking renders access very slow. As in many use cases network access could be almost as fast as the drive access or even faster (e.g. on an EC2 instance), it might make sense to make such caching optional, or even not enabled by default at all so we could use non-patched fsspec with default installation of datalad-fuse.

So let's add --caching none,ondisk with none as default, and provide documentation in e.g. README to say that patched version of fsspec is needed for ondisk.

NB while coming up with interface, might be worth considering that later we might make caching support configurable as well, so "ondisk" might not be descriptive enough. But we could think about it whenever we get there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants