Crashes on empty directories when storing data in NFS

[sh0rez]

Hi, after pulling the latest version of the docker image, victoria was unable to come up again:

cannot open a storage at victoria-metrics-data with retention period 120 months:
cannot open indexdb tables at "/victoria-metrics-data/indexdb":
cannot open curr indexdb table at "/victoria-metrics-data/indexdb/15A73A586D904DA0":
cannot open indexDB "/victoria-metrics-data/indexdb/15A73A586D904DA0":
cannot open table parts at "/victoria-metrics-data/indexdb/15A73A586D904DA0":
cannot open part "/victoria-metrics-data/indexdb/15A73A586D904DA0/90_1_15A73A586EF234CE":
cannot parse path to part:
cannot read "/victoria-metrics-data/indexdb/15A73A586D904DA0/90_1_15A73A586EF234CE/metadata.json":
open /victoria-metrics-data/indexdb/15A73A586D904DA0/90_1_15A73A586EF234CE/metadata.json:
no such file or directory

[sh0rez]

Okay, I was able to resolve it. Because of the empty NFS directories, which were not removed because of #61, there were a lot of bloating empty directories (100+). As there were no metadata.json inside of these, victoria crashed.

Although this should not happen in recent versions anymore, empty directories should probably cleaned up on startup as well.

[valyala]

@sh0rez , could you confirm the following for the recent version of VictoriaMetrics?

• It doesn't create empty NFS directories when accepting new data
• It doesn't emit error logs related to directory deletion
• It restarts without issues

[sh0rez]

@sh0rez , could you confirm the following for the recent version of VictoriaMetrics?

• It doesn't create empty NFS directories when accepting new data
• It doesn't emit error logs related to directory deletion
• It restarts without issues

These are all correct, as long as the dataset is not from the old version. IF the broken version created the empty folders, the new version crashes as well.

So maybe just clean up empty dirs on startup in case they exist?

[valyala]

These are all correct, as long as the dataset is not from the old version

This is great news! As for the data from old version, it is expected that new versions crash on empty dirs. This helps fixing the problem as it appears instead of hiding it until it explodes.

So maybe just clean up empty dirs on startup in case they exist?

Exactly.

Then closing the issue. Feel free re-opening it if new version will start failing on NFS disks.

[valyala]

See also #162