Outdated free/used disk space values

[xfilokolo]

Hi. This is a problem that has been around for a long time. The web GUI and CLI dashboard show outdated values of how much free disk space is available.

Example:

• In the dashboard, I see "8GB of free space"
• Restarts the node with increased free space
• I still see "8GB of free space" in the dashboard (same in web and CLI dashboard)

[phpbg]

May be related to : #3783

[zeebo]

So because you may be storing blobs on a hard drive with other things stored on it, df is not useful for determining free space. That only leaves us walking the filesystem and counting all of the files, which is an expensive operation.

Because that operation is so expensive, we have a cache that persists its understanding of how much space is used, and periodically walks the filesystem to stay as fresh as possible. If you give it long enough, it should calculate how much is used after it starts.

I'm not sure that's what's going on here, though. What exactly did you do for the "restarting the node with increased free space" step? We use the storage.allocated-disk-space configuration value to determine how much "free space" there is, and double check that against the reported value of df to ensure it's reasonable. Did you bump that configuration value when you restarted the node?

[xfilokolo]

Exactly. I turned off the node, increased the amount of space allocated to it, and then started it again. I will add that I used rm docker with every change.

[MaxMaximus]

I think dashboard shows not outdated values but simply wrong values.
Dashboard on my full (completely filled) nodes still shows there is about 10-20 GB available. And keep showing it after restart and long periods of time.

Also there is a significant discrepancy between values shown in Dashboards and values reported in logs (which actually used for decisions making). For example there is that i see in dashboard

31 GB available space reported by dashboard
And that i see if i check logs of this same node at the same time:

2020-05-09T17:49:55.472+0300	INFO	piecestore	upload started	{"Piece ID": "SB3R2MPIIVQGR3LEMNBJYGOWJK2GN545RJWYFTQ7ZP34RXXHEH5A", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 7759797120}
2020-05-09T17:49:56.203+0300	INFO	piecestore	upload started	{"Piece ID": "OUBQNZGYDKFU6AELCGGAVRMLJIHHHITZPMR6LC4F5IBZRFYMKRDA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 7759797120}
2020-05-09T17:49:57.087+0300	INFO	piecestore	upload started	{"Piece ID": "2AMJD4UWUUWKYQFR2XHVNW66J5XT4NMWZ4FVJQJTENHYINVU2S4Q", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 7759797120}
2020-05-09T17:49:57.389+0300	INFO	piecestore	upload canceled	{"Piece ID": "OUBQNZGYDKFU6AELCGGAVRMLJIHHHITZPMR6LC4F5IBZRFYMKRDA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"} 

Only 7.7 GB of space available reported in the logs.
And when value in the log reach the zero - node stops accepting upload. While dashboard keep showing > 20 GB of space available, but node does not accepts uploads.

I did a little data digging and looks like i have found source of discrepancy:
1 - dashboard counts only space occupied by "blob" folder (primary data pieces storage): so available space there calculated as limit set in config minus size of "blobs" folder
2 - but actual value used counts for all folders used for storage including temporal storage (blobs + trash + temp + garbage folders). So available space calculated as limit set in config minus size of all data folders.
Because for my node limit is set to storage.allocated-disk-space: 2.9 TB
And current blobs folder size is 2.87 TB = dashboard reports 30 GB available (2.9 - 2.87 TB = ~30 GB)
But total STORJ data folder size is 2.893 TB, so logs report correct value (2.9 TB - 2.893 TB = 7 GB)
And dashboard values are simply wrong because it miss ~23 GB of additional space currently occupied by "temp" "garbage" folders.

P.S.
Looks like it can be related to #3783

[MaxMaximus]

Update.
Now node is full again. And uploading stopped due to out of space. There are last uploads:

2020-05-09T20:38:18.075+0300	INFO	piecestore	uploaded	{"Piece ID": "WJGTEQ7PW4WM5QYKLZC5HELMJ7HNZ5KFS4XFAY2PVAYXX2VDYVFA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-05-09T20:38:18.264+0300	INFO	piecestore	upload started	{"Piece ID": "RJPFQVUCDRCNV3GKT56N3J25TMKJKGHUIVKETC3EZ2F3WWC5ADGQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT", "Available Space": 95587456}
2020-05-09T20:38:18.467+0300	INFO	piecestore	uploaded	{"Piece ID": "LVKMN35VKXSDNIPM3MVIJEUDPNFMOTX2CRKZBXIZG2FTIEBHGC3A", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-05-09T20:38:19.621+0300	INFO	piecestore	upload canceled	{"Piece ID": "YAVVFP4ZT4IB6IPNGVZ5N5SDDUYRSAU7RQ536D3FPGZWVJPRNXKA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-05-09T20:38:20.281+0300	INFO	piecestore	uploaded	{"Piece ID": "RJPFQVUCDRCNV3GKT56N3J25TMKJKGHUIVKETC3EZ2F3WWC5ADGQ", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}
2020-05-09T20:38:21.200+0300	INFO	piecestore	uploaded	{"Piece ID": "SY4IFCC5DO67QJMQ35CIBY7ED4S7WRNHW3YNP3EHK3O37SRV56CA", "Satellite ID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "Action": "PUT"}

Just few dozen MB left before stop.
But that i see in the Dashboard (both right after uploads stops and hour later and after node restart:

Disk Space Remaining
24.17GB

Same 23-24 GB shown as Space Remaining while actually it already occupied by "trash" (23 GB currently, and others <1 GB).
Size of "blob" folder is 2 876 311 565 312 bytes. storage.allocated-disk-space: 2.9 TB