New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
UrlStore documentation of cache behavoir (when combining with MFS) #6928
Comments
Note: you'll still need to download the data once (throwing it away immediately) to chunk/hash it.
Under the covers, we create "fake" blocks in a "fake" datastore that refer to slices of files that can be found at known URLs. We won't ever end up caching those blocks locally because, from IPFS's perspective, we already have them. Are you interested in writing this up in the "docs/experimental-features.md" file? |
Sure :)
Thanks!
Yes, feel free to assign me :) |
Question: What happens if I run something like:
and some parts of the file match a file already stored? Will the blocks be dropped, and fetched on demand from the URL? |
They'll stay and we won't fetch them from the remote server. Also, I wasn't quite correct when I said:
If you call |
Thanks! Very interesting! |
@Stebalien what happens if I add a file twice, both times with --nocopy and different URLs? Will the URL be saved as two locations, or will the old URL be overwritten? |
I believe it the last-used URL will win, but you'd have to check. |
Hey guys there's currently very little documentation available about the experimental function "UrlStore".
I understand that there's no local data cached while adding the file to IPFS, rather a link is pointing to an URL where the data is fetched on demand.
I like to combine this feature with MFS for easy altering of the file/folder structure, while the HTTP server still holds the data.
The response time of a
ipfs files cp
of a CID stored with UrlStore suggests, that the MFS lazy-mounts the new CID to the local MFS and not pulls the content to the local cache.The question arises if the
--nocopy
flag is a general 'no local caching' flag, or if my local cache will start caching the files when they are requested from the IPFS because I've added it to the MFS. While the inclusion of the CID in the MFS will hinder any garbage collection attempts of those files from the cache, basically resulting in a rising cache usage until my local storage is used up (http server holds like 2 magnitudes more storage than I have locally).Best regards
Ruben
The text was updated successfully, but these errors were encountered: