after disk full was resolved, InfluxDB2 continues to write just to RAM

[cholzer79]

I was asked to post this here.

I’ve been running influxdb 2.7 on my Ubuntu 22.04.4 LTS VM for over 6 months without any issues.

Last week the system ran out of diskspace which I noticed when I received an alert about high RAM usage (I suspect influxdb was writing just to RAM for 2 days).

I then expanded the disk to resove the problem.

However whenever I restarted the VM, or just restarted the influxdb service, influxdb lost all data it had logged since the last start.
It appears as if influxdb continued to just log data to RAM instead of the disk as if the disk was still full.

But “systemctl status influxdb” as well as “journalctl -u influxdb.service” showed no errors.

My solution then was to restore a VM backup from the day before the disk was full.
I then expanded the disk like before and influxdb is writing data to the disk just fine now - restarts are no issue.

It looks like running out of diskspace upset influxdb, which caused it to only write to RAM even after the diskspace issue was resolved.

[philjb]

Thanks for your report - It appears the issue has been resolved for yourself? yes?

Did you happen to capture any logs while you suspected that influxd was only writing to memory? I would not expect influx to continue to work after the disk runs out. The last wal file or tsm file influxd tried to write when the disk filled could have been truncated and thus corrupt. That situation might have needed the restore from a backup as you did to resolve/remove any problematic files on disk.

For a future reader finding this issue, I would do this procedure:

• Realize the disk is full.
• Shutdown influxd
• expand/increase available disk
• restart influxd and watch the logs and file system to ensure that influxd is writing wal files and snapshots into new tsm files. Check write requests for errors. If there are no errors or warning in the logs and wal/tsm files are appearing, influxd should be operating correctly (try a query too).

As it seems your issue is resolved, I'm going to close this ticket. Please reopen and provide more detail if you still have an issue.