fix storage limits

[whyrusleeping]

we currently have code that is supposed to limit the size of an fsrepo, but i'm fairly certain it doesnt actually work. We need to go through and fix (and test this).

This is also (I beleive) related to the auto-gc code which i'm not sure actually works either.

Code links:

• https://github.com/ipfs/go-ipfs/blob/master/repo/fsrepo/fsrepo.go#L548
• https://github.com/ipfs/go-ipfs/blob/master/repo/config/datastore.go#L14

Notes:

• need to keep approximate repo size so we don't have to constantly recalculate

[lanzafame]

@whyrusleeping if this is still open, I would love to take a crack at this?

[whyrusleeping]

@lanzafame I don't believe this has been resolved. At the very least, we need to test it. If the test proves that its broken, then we need to fix it. Go ahead and give it a shot :)

[lanzafame]

@whyrusleeping So I re-enabled the repo-gc-auto tests, and they still fail.
output:

*** test/sharness/t0082-repo-gc-auto.sh ***
/Users/lanzafame/dev/src/github.com/ipfs/go-ipfs/test/sharness
ok 1 - ipfs init succeeds
ok 2 - prepare config -- mounting
ok 3 - generate 2 600 kB files and 2 MB file using go-random
ok 4 - set ipfs gc watermark, storage max, and gc timeout
ok 5 - 'ipfs daemon' succeeds
ok 6 - api file shows up
ok 7 - set up address variables
ok 8 - set swarm address vars
ok 9 - 'ipfs daemon' is ready
ok 10 - adding data below watermark doesn't trigger auto gc
not ok 11 - adding data beyond watermark triggers auto gc
#
#	    HASH=`ipfs add -q 600k2` &&
#	    ipfs pin rm -r $HASH &&
#	    go-sleep 40ms &&
#	    DU=$(disk_usage "$IPFS_PATH/blocks") &&
#	    if test $(uname -s) = "Darwin"; then
#	      test "$DU" -lt 1400  # 60% of 2MB
#	    else
#	      test "$DU" -lt 1000000
#	    fi
#
not ok 12 - adding data beyond storageMax fails
#
#	  test_must_fail ipfs add 2M 2>add_fail_out
#
not ok 13 - ipfs add not enough space message looks good
#
#	  echo "Error: file size exceeds slack space allowed by storageMax. Maybe unpin some files?" >add_fail_exp &&
#	  test_cmp add_fail_exp add_fail_out
#
ok 54 - periodic auto gc stress test
ok 55 - 'ipfs daemon' is still running
ok 56 - 'ipfs daemon' can be killed
# failed 32 among 56 test(s)

[lanzafame]

I have been able to solve 12, but am still digging into 11.

[lanzafame]

@whyrusleeping I have been digging around all gc related areas of the code base and am struggling to figure out where the daemon should be triggering auto gc due to adding data beyond the watermark? Any ideas would be greatly appreciated 👍

[whyrusleeping]

@lanzafame hrm... I think @kevina or @Kubuxu might have an idea

[kevina]

@whyrusleeping I am not that familiar with the code, should the auto-gc be triggered at all? I thought we removed that code.

[whyrusleeping]

I'm pretty sure its still there, i believe @lgierth has it running on the gateways...

[ghost]

• StorageMax
  An upper limit on the total size of the ipfs repository's datastore. Writes to
  the datastore will begin to fail once this limit is reached.

This part in docs/config.md is currently inaccurate - the datastore will generally accept writes until the disk is full. The StorageMax value is used for:

1. printing it in ipfs repo stat
2. checking whether GC watermark is hit and GC should run

So you need to run with --enable-gc for StorageMax to do anything at all. When GC is active, it's gonna check repo size every GCPeriod, and trigger if it's more than (StorageMax * 100 / GCWatermark).

[ghost]

So eventually we should make StorageMax a hard limit on the diskspace used, as it was meant in the beginning, but I think we should defer that to next year, since proper accounting of the diskspace used (without constantly scanning the repo) will actually be a bit tricky, and that's just one ball too much in the air right now :)

@lanzafame do you think you could come up with better wording for the StorageMax docs though?

[lanzafame]

@lgierth Just checking that I understand correctly, with the --enable-gc flag set, the StorageMax value is used with the GCWatermark value to determine whether a garbage collection should be run at every GCPeriod?

[ghost]

@lanzafame yep correct!

[kallisti5]

Just a smaller user note on this one. We're working up instructions for folks to mirror our rather large software repos on IPFS.
IPFS-Desktop ships with --enable-gc and a StorageMax of 10GB.

This creates an interesting situation. If the pin fails to finish, IPFS needs to "re-download" all chunks as it will garbage collect them before the next run. This is super frustrating.

[aschmahmann]

@kallisti5 note that almost any solution to the storage limits here is going to result in a problem in your scenario. This issue is mostly around StorageMax not being intuitive in that you can store more than the StorageMax. There are a many different models for resolving this including:

1. Making StorageMax a hard cap
2. Replacing StorageMax with CacheHighWater which indicates how much data you're willing to keep in cache aside from any pinned data
3. Having both a StorageMax and a CacheHighWater so that storage is always controlled but the amount of cached data is limited as well

No matter how you slice it though in order to protect users' space there will exist some scenario where after a partial download automatic GC that could end up deleting data that isn't protected from GC. It sounds like what you are looking for is a best-effort-pin (#3121, but there may be other issues too) wherein you are trying to pin the data but don't need (or even want) all of it, for example a best effort wikipedia pin would protect any wikipedia data from GC but without downloading all of it.

IIRC you can achieve this today with MFS by adding the root to MFS. In your case this would translate to adding the root to MFS and the pinning the rest of the data. However, in your case if a user has StorageMax set at 10GB and you want them to download 20GB they should just be increasing their StorageMax.