[storagenode] Monitor and Enable lazyfilewalker by default

[profclems]

Describe the feature or problem you'd like to solve

The lazyfilewalker runs the used-space and garbage collection filewalkers as a low priority subprocess. It came up as a solution to #5349, where these filewalkers run with the same I/O priority as customer downloads.

The lazyfilewalker is disabled by default.

What needs to be done:

• Enable the lazyfilewalker on a production node. Maybe ask SNOs to volunteer on the forum?
• Monitor and collect profiles and monkit logs
• Enable the lazyfilewalker by default

[profclems]

Blocked by #5900

[iglesiasbrandon]

We will ask the SNO community to volunteer to test out this functionality to determine if they have any issues.

[profclems]

I'm running v1.80.1-rc from the QA version server with the fix for the broken pipe error:

docker logs storagenode | grep "filewalker completed"
2023-05-25T16:19:50.566Z	INFO	lazyfilewalker.used-space-filewalker.subprocess	used-space-filewalker completed	{"process": "storagenode", "satelliteID": "12rfG3sh9NCWiX3ivPjq2HtdLmbqCrvHVEzJubnzFzosMuawymB", "process": "storagenode", "piecesTotal": 4639744, "piecesContentSize": 4638720}
2023-05-25T16:19:50.838Z	INFO	lazyfilewalker.used-space-filewalker.subprocess	used-space-filewalker completed	{"process": "storagenode", "satelliteID": "1GGZktUwmMKTwTWNcmGnFJ3n7rjE58QnNcRp98Y23MmbDnVoiU", "piecesTotal": 846728704, "piecesContentSize": 845540864, "process": "storagenode"}
2023-05-25T16:19:51.095Z	INFO	lazyfilewalker.used-space-filewalker.subprocess	used-space-filewalker completed	{"process": "storagenode", "satelliteID": "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs", "process": "storagenode", "piecesTotal": 5286875136, "piecesContentSize": 5284692992}
2023-05-25T16:19:51.214Z	INFO	lazyfilewalker.used-space-filewalker.subprocess	used-space-filewalker completed	{"process": "storagenode", "satelliteID": "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6", "process": "storagenode", "piecesTotal": 1339260416, "piecesContentSize": 1338493952}
2023-05-25T16:19:52.155Z	INFO	lazyfilewalker.used-space-filewalker.subprocess	used-space-filewalker completed	{"process": "storagenode", "satelliteID": "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S", "piecesContentSize": 6823678720, "process": "storagenode", "piecesTotal": 6847349504}
2023-05-25T16:19:52.200Z	INFO	lazyfilewalker.used-space-filewalker.subprocess	used-space-filewalker completed	{"process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "process": "storagenode", "piecesTotal": 135126016, "piecesContentSize": 135091712}
2023-05-25T16:19:52.244Z	INFO	lazyfilewalker.used-space-filewalker.subprocess	used-space-filewalker completed	{"process": "storagenode", "satelliteID": "12tRQrMTWUWwzwGh18i7Fqs67kmdhH9t6aToeiwbo5mfS2rUmo", "process": "storagenode", "piecesTotal": 1200128, "piecesContentSize": 1198080}

From the logs, lazyfilewalker for the used-space calculation completes successfully, rightly calculates and returns the results to the main process.

[profclems]

Logs from an instance of the gc-filewalker, also completed successfully:

05:06:14 root@testnode ~ → docker logs storagenode | grep lazyfilewalker.gc-filewalker
2023-05-25T17:00:22.139Z	INFO	lazyfilewalker.gc-filewalker	starting subprocess	{"process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2023-05-25T17:00:22.142Z	INFO	lazyfilewalker.gc-filewalker	subprocess started	{"process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}
2023-05-25T17:00:22.166Z	INFO	lazyfilewalker.gc-filewalker.subprocess	Database started	{"process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "process": "storagenode"}
2023-05-25T17:00:22.168Z	INFO	lazyfilewalker.gc-filewalker.subprocess	gc-filewalker started	{"process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "process": "storagenode", "createdBefore": "2023-04-18T18:08:51.963Z", "bloomFilterSize": 239182}
2023-05-25T17:00:22.172Z	INFO	lazyfilewalker.gc-filewalker.subprocess	gc-filewalker completed	{"process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "process": "storagenode", "piecesCount": 67, "piecesSkippedCount": 0}
2023-05-25T17:00:22.176Z	INFO	lazyfilewalker.gc-filewalker	subprocess finished successfully	{"process": "storagenode", "satelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE"}

[profclems]

Forum post: https://forum.storj.io/t/enabling-the-lazyfilewalker/22966?u=clement

[iglesiasbrandon]

hey @profclems there are alot of comments from the community on that post, can you take a look and respond to them please?

[profclems]

I created a new issue from a bug on windows reported on the forum post #6006

[profclems]

Blocked by #6006

[iglesiasbrandon]

#6006 has been merged but not deployed so this ticket is on pause

[storj-gerrit]

Change storagenode/pieces: enable lazyfilewalker by default mentions this issue.