Data added via ipfs add --to-files=/example --pin=false gets garbage collected.
#9553
Comments
kevincox
added
kind/bug
Unable to reproduce, user data is not lostThank you for filling this @kevincox,
But there is something unexpected!I suspect you've also seen some CIDs getting garbage-collected during step (3) Investigating the ephemeral block that got GC'dWhen represented as CIDv0 (or CIDv1 with dag-pb): $ ipfs cid format -v 0 --mc dag-pb bafkreiakaltuk5nrwxeonlmztefw3qcjj7s6mxlilyvzfedp7zh6q5fc2a
QmP1jMW7thNHXB2hiUPmKUqa8mJvBYofuVTd5NVQzsH4e3QmP1jMW7thNHXB2hiUPmKUqa8mJvBYofuVTd5NVQzsH4e3.car.zip Exploring the CID, we see it is a wrapper directory that included the imported file, but is no longer used. 👉 So it seems there is no user data loss, but I think you've found that @kevincox are you able to check and confirm this is what you are seeing? |
|
That is not the case. It is definitely occurring that the directory added to MFS no longer is complete, so user data is lost. Unfortunately my previous reproductions were the "wait a while" case. In my case it was a directory that was about 75MiB large. I'll see if I can reproduce with an explicit GC. |
|
I can't seem to reproduce with an explicit GC. I'll share more of the context in case it helps. I have a backup script that snapshots a directory to IPFS. It is basically this. id=$(date -u '+%FT%TZ')
# Add
ipfs --offline add --to-files=/backups/something/$id --pin=false -rHQ -s rabin-2048-65536-131072 -- /some/directory
# Update latest pointer.
ipfs --offline files rm -r /backups/something/latest || true
ipfs --offline files cp /backups/something/$id /backups/something/latest
# Maybe cleanup old backups. Actual logic is smarter and **does not** always delete the oldest first.
ipfs --offline files rm -r -- $(ipfs files ls /backups/something | head -n1)
# Calculate total space usage. THIS STEP FINDS MISSING BLOCKS
ipfs --offline files stat /backups/something
# Pin remotely.
ipfs pin remote add ...
# IPNS publish.
ipfs name publish --allow-offline --ipns-base b58mh --key something --lifetime 8784h --quieter $cid
# Save published CID to avoid GC.
ipfs --offline files rm -r /backups/.last-published-something
ipfs --offline files cp /backups/something /backups/.last-published-somethingI've used this script for a long time. It originally did an When I investigated which subdirectory of MFS had the missing blocks it was never the latest one. It was always a directory a few backups ago. One that had previously been checked to be complete and was completely unchanged. The path was never removed or even moved since the validation succeeded. |
|
I poked at this a bit in that direction, run If there is a problem, it is something more nuanced – we need to find a way to reproduce it first. @kevincox any chance you can create a minimal |
|
Unfortunately I'm not really sure that I can. I would need to understand what is happening more to be able to reproduce in a more minimal setting. I tried simulating the workload I had with random data and didn't succeed. I tried sharing some blocks between "backups" and also couldn't reproduce. It seems that anything I strip down removes the magic condition. I think it is very likely that it is something nuanced. I've included a cleaned up version of my backup script below in case it triggers any ideas but I'm afraid that we may be unable to debug this until someone else has this issue and we can try to spot the similarities. root="/backups/something"
latest=$root/latest
new=$root/$(date -u '+%FT%TZ')
ipfs --offline files mkdir -p $root
### BROKEN
ipfs --offline add --to-files=$new --pin=false '-rHQ' '-s' 'rabin-2048-65536-131072' '--' '/var/lib/something/'
read latest_cid latest_flat_size < <(ipfs --offline files stat --format '<hash> <cumulsize>' $new)
### BROKEN END
### WORKING
ipfs --offline add --to-files=$new --pin=true '-rHQ' '-s' 'rabin-2048-65536-131072' '--' '/var/lib/something/'
read latest_cid latest_flat_size < <(ipfs --offline files stat --format '<hash> <cumulsize>' $new)
ipfs --offline pin rm $latest_cid
### WORKING END
last=$(ipfs --offline files stat --hash $latest || true)
if [[ $last == "" ]]; then
diff=new
elif [[ $last != $latest_cid ]]; then
diff=updated
else
diff=unchanged
fi
if [[ $diff != unchanged ]]; then
ipfs --offline files rm -r $latest || true
ipfs --offline files cp /ipfs/$latest_cid $latest
fi
ipfs --offline files ls $root \
| grep -v '^latest$' \
| pruner 'P1D=14' 'P1M=6' '10' \
| awk -v root=$root/ '{print root $0}' \
| xargs --verbose -L1 -r \
ipfs --offline files rm -r --
read root_cid root_flat_size < <(ipfs --offline files stat --format '<hash> <cumulsize>' $root)
key="backups-something"
ipfs key gen $key || true
ipns=$(ipfs name publish \
--allow-offline \
--ipns-base b58mh \
--key "$key" \
--lifetime $((366*24))h \
--quieter \
$root_cid)
ipfs --offline files rm -r "/backups/.last-published-something" || true
ipfs --offline files cp "$root" "/backups/.last-published-something"
root_usage=$(ipfs --offline dag stat --progress=false $root_cid)
eval 'root_usage=${${root_usage##Size: }%,*}'
latest_usage=$(ipfs --offline dag stat --progress=false $latest_cid)
eval 'latest_usage=${${latest_usage##Size: }%,*}' |
|
Ok, I switched back to no pinning and this happened again.
The detection was probably quite late in this case because I had some problems with my pinning services so the full |
|
Had this again very quickly. This time two paths were missing.
|
|
I saw this a few times, but due to paths getting removed before I investigated it was hard to get a good trace. The most recent time was very fast, about an hour after the data was added it was lost.
This is actually the same backup run (the stat took a long time with a cold cache) so very little was done between the add and the loss. |
guseggert
added
need/maintainer-input
|
I just tried this again with kubo-0.20.0 and the issue recurred quickly. |
|
2023-08-03 maintainer conversation: we're not prioritizing this currently because don't know how to reproduce the problem, and if there is a problem, it's likely in MFS (and we need to product work around data onboarding and MFS). |
Checklist
Installation method
third-party binary
Version
Config
{ "API": { "HTTPHeaders": { "Access-Control-Allow-Origin": [ "https://ipfs.example" ] } }, "Addresses": { "API": "/unix/run/ipfs-api.sock", "Announce": [], "AppendAnnounce": null, "Gateway": "/unix/run/ipfs-gateway.sock", "NoAnnounce": [ "/ip4/10.0.0.0/ipcidr/8", "/ip4/100.64.0.0/ipcidr/10", "/ip4/169.254.0.0/ipcidr/16", "/ip4/172.16.0.0/ipcidr/12", "/ip4/192.0.0.0/ipcidr/24", "/ip4/192.0.2.0/ipcidr/24", "/ip4/192.168.0.0/ipcidr/16", "/ip4/198.18.0.0/ipcidr/15", "/ip4/198.51.100.0/ipcidr/24", "/ip4/203.0.113.0/ipcidr/24", "/ip4/240.0.0.0/ipcidr/4", "/ip6/100::/ipcidr/64", "/ip6/2001:2::/ipcidr/48", "/ip6/2001:db8::/ipcidr/32", "/ip6/fc00::/ipcidr/7", "/ip6/fe80::/ipcidr/10" ], "Swarm": [ "/ip4/0.0.0.0/tcp/4001", "/ip6/::/tcp/4001", "/ip4/0.0.0.0/udp/4001/quic", "/ip6/::/udp/4001/quic" ] }, "AutoNAT": {}, "Bootstrap": [ "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ", "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ", "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN", "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa", "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb", "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt" ], "DNS": { "Resolvers": null }, "Datastore": { "BloomFilterSize": 0, "GCPeriod": "1h", "HashOnRead": false, "Spec": { "mounts": [ { "child": { "path": "blocks", "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2", "sync": true, "type": "flatfs" }, "mountpoint": "/blocks", "prefix": "flatfs.datastore", "type": "measure" }, { "child": { "compression": "none", "path": "datastore", "type": "levelds" }, "mountpoint": "/", "prefix": "leveldb.datastore", "type": "measure" } ], "type": "mount" }, "StorageGCWatermark": 90, "StorageMax": "10GB" }, "Discovery": { "MDNS": { "Enabled": false, "Interval": 10 } }, "Experimental": { "AcceleratedDHTClient": false, "FilestoreEnabled": false, "GraphsyncEnabled": false, "Libp2pStreamMounting": false, "P2pHttpProxy": false, "StrategicProviding": false, "UrlstoreEnabled": false }, "Gateway": { "APICommands": [], "HTTPHeaders": { "Access-Control-Allow-Headers": [ "X-Requested-With", "Range", "User-Agent" ], "Access-Control-Allow-Methods": [ "GET" ], "Access-Control-Allow-Origin": [ "*" ] }, "NoDNSLink": false, "NoFetch": false, "PathPrefixes": [], "PublicGateways": null, "RootRedirect": "", "Writable": false }, "Identity": { "PeerID": "REDACTED" }, "Internal": {}, "Ipns": { "RecordLifetime": "", "RepublishPeriod": "", "ResolveCacheSize": 128 }, "Migration": { "DownloadSources": null, "Keep": "" }, "Mounts": { "FuseAllowOther": false, "IPFS": "/ipfs", "IPNS": "/ipns" }, "Peering": { "Peers": null }, "Pinning": { "RemoteServices": { // REDACTED } }, "Plugins": { "Plugins": null }, "Provider": { "Strategy": "" }, "Pubsub": { "DisableSigning": false, "Router": "" }, "Reprovider": { "Interval": "12h", "Strategy": "all" }, "Routing": { "Methods": null, "Routers": null, "Type": "dhtclient" }, "Swarm": { "AddrFilters": [ "/ip4/10.0.0.0/ipcidr/8", "/ip4/100.64.0.0/ipcidr/10", "/ip4/169.254.0.0/ipcidr/16", "/ip4/172.16.0.0/ipcidr/12", "/ip4/192.0.0.0/ipcidr/24", "/ip4/192.0.2.0/ipcidr/24", "/ip4/192.168.0.0/ipcidr/16", "/ip4/198.18.0.0/ipcidr/15", "/ip4/198.51.100.0/ipcidr/24", "/ip4/203.0.113.0/ipcidr/24", "/ip4/240.0.0.0/ipcidr/4", "/ip6/100::/ipcidr/64", "/ip6/2001:2::/ipcidr/48", "/ip6/2001:db8::/ipcidr/32", "/ip6/fc00::/ipcidr/7", "/ip6/fe80::/ipcidr/10" ], "ConnMgr": { "GracePeriod": "20s", "HighWater": 900, "LowWater": 600, "Type": "basic" }, "DisableBandwidthMetrics": false, "DisableNatPortMap": true, "RelayClient": {}, "RelayService": {}, "ResourceMgr": {}, "Transports": { "Multiplexers": {}, "Network": {}, "Security": {} } } }Description
Problem
ipfs add --to-files=/example --pin=false /some/new/dataipfs dag stat --offline $(ipfs --offline files stat --hash /example)Expected results: Files in MFS are expected to be kept on the node so this command should succeed.
Actual result: "Error: error traversing DAG: block was not found locally (offline): ipld: could not find QmatWhxBmsvpPCAuzryVeQygiWntsi4bZHm1Gy7PnakPEr"
Workaround
As a workaround you can create then remove a pin, this basically defeats the point of adding
--to-filesbut avoids this issue.Caveats:
ipfs add --to-files=/example --pin=true /some/new/data ipfs pin rm $(ipfs --offline files stat --hash /example)Even thought this leaves you in the same end state from the user perspective (save the caveats mentioned above) this does seem to resolve the internal state such that the MFS "implicit pinning" avoids the content from being collected.
Context
ipfs add --to-filesipfs add --to-files=/path/in/mfs #8504cc @lidel who implemented this feature.
Other questions
Do you need to pass
--pin=falsewhen using--to-files? Theipfs help adddocs seem to claim that it is still true by default. However when adding to MFS you shouldn't need the pin. With the implicit pin the--to-filesflag seems to just be a shortcut foripfs files cp /ipfs/$(ipfs add /some/new/data) /example. Maybe the docs can be clarified here.The text was updated successfully, but these errors were encountered: