You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
ATM local filesystem caching of accessed content is non-optional. To make fsspec work multi-threaded, heavy locking was introduced and we are using custom branch with that. That locking renders access very slow. As in many use cases network access could be almost as fast as the drive access or even faster (e.g. on an EC2 instance), it might make sense to make such caching optional, or even not enabled by default at all so we could use non-patched fsspec with default installation of datalad-fuse.
So let's add --caching none,ondisk with none as default, and provide documentation in e.g. README to say that patched version of fsspec is needed for ondisk.
NB while coming up with interface, might be worth considering that later we might make caching support configurable as well, so "ondisk" might not be descriptive enough. But we could think about it whenever we get there.
The text was updated successfully, but these errors were encountered:
ATM local filesystem caching of accessed content is non-optional. To make fsspec work multi-threaded, heavy locking was introduced and we are using custom branch with that. That locking renders access very slow. As in many use cases network access could be almost as fast as the drive access or even faster (e.g. on an EC2 instance), it might make sense to make such caching optional, or even not enabled by default at all so we could use non-patched fsspec with default installation of datalad-fuse.
So let's add
--caching none,ondisk
withnone
as default, and provide documentation in e.g. README to say that patched version of fsspec is needed forondisk
.NB while coming up with interface, might be worth considering that later we might make caching support configurable as well, so "ondisk" might not be descriptive enough. But we could think about it whenever we get there.
The text was updated successfully, but these errors were encountered: