Data loss when system is unplugged

[sdumetz]

I have a product that can sometimes be unplugged by mistake when running. Nothing I can do about customers not being careful.

When this happens, I sometimes have my DB completely erased.
This is similar to #92, except I used 0.10 where the issue is resolved, and it does not seem to happen on load (I only load my Datastore on startup).

I tried to reproduce the problem but can't seem to find a pattern on why the Datastore is sometimes erased and sometimes not.

looks like a power failure during an update can cause a file corruption.

can I make some sort of security over this using NeDB? I don't mind loosing the last couple of updates as long as my "long term data" is intact.

[szwacz]

Could it be that appending to file is also affected by this issue?
How the database file looks after such incident? The file is not-present, or present and empty?

[sdumetz]

can't see as I could'nt safely reproduce the bug in lab conditions, and on production, DB is immediately created if missing.

As far as I know, neDB doesn't give feedback wether db already exists on new Datastore("filename") call?

I personally see 2 options :

• find a way log load results this to have tangible data on this issue.
• reproduce the bug without having to unplug my system multiple times while running.

Do you have an idea on how to do any of those?

[RobertWHurst]

I originally thought that this might be due to the compaction process, but I don't think so.

Lets say your datastore is named datastore.nedb. When compaction happens, NeDB will rename your datafile ~~datastore.nedb to protect it. It will then write data from memory to a new temp datafile named ~datastore.nedb. Once complete, the temp file is renamed datastore.nedb replacing the original file. The old datastore, ~~datastore.nedb, is then deleted. If a failure occurs during this process, the backup ~~datastore.nedb will remain, and the program is started again, and NeDB is initialized, it will find the ~~datastore.nedb file and recover it, renaming it datastore.nedb.

The issue could happen during an insert, but I doubt it. When data is inserted into the datastore, the persistence layer appends the newly inserted document directly to the datastore.nedb. It does not make a backup or temp file, and I'd imagine this is strictly because of performance. You don't want to throttle the OS by creating and destroying files each write. I'm guessing this might be the issue because I'm not actually sure what node's fs.appendFile method does when writing to a file. I can say that if as long as the file can be read as text, even a corrupt document will not stop NeDB from loading a data file. If any document within the file cannot be deserialized it's simply skipped. Every other document in the file would be loaded.

Just curious, what are you running this process on, and why is it randomly unplugged while running?

[sdumetz]

it's running on a display - think of it as an all in one pc - my company is selling (don't wanna make an ad here so just check my profile if curious).

The thing is : customers aren't always careful and they sometimes pull on the power cord. When this happen, I do prefer to have an unplugged device than a broken one (ie falling because of this), so I can't simply solder my power cord to the product.

I didn't enabled autocompaction, and because of my app structure, it would take the product to be unplugged exactly at the moment I do my new Datastore("filename") call to fail because of this.

I'll look into node fs.appendFile to see if I can find anything.

[RobertWHurst]

@klepthys I had to look. Now that's a pretty bad ass product.

[sdumetz]

After searching around a bit, it seems there is no real way to ensure further integrity on writes.
besides applying safer options to my filesystem, I don't know if there is an opportunity to make some sort of backup capability on NeDB. I'll look into a solution for my particular case and come back if I think it can reasonably be added as an option.

Closing this issue as there is probably no bug in NeDB here.

[louischatriot]

That's indeed a badass product, I love to see NeDB being part of it !
Concerning your issue, it does indeed seem to have to do with the underlying filesystem, and possibly non atomic writes. The strange part here is that when loading a data file, NeDB loads documents line by line and automatically discards any line that doesn't have the right format but not the rest. So a bad append should in theory not have impact on the rest of the data, but I expect some exoctic filesystems to display such strange behaviors ...

[zevero]

Yesterday my nodejs server was rebooted and two independent db files ended up with size 0!!!

Luckily we do three backups per day.

But it sure was strange ...

[louischatriot]

@zevero do you have more data ? The loading system is designed to protect your data against this kind of behavior. What OS and FS are you using ? Have you been able to reproduc ? With what steps ?

[ejok]

I also ran into this problem. I'm running NeDB in a system where by design the system is powered off from a switch. Occasionally when the system booted up again, the database file is 0 bytes. I did some digging and found out that ext4 file system, that we are using, has had an issue that caused data loss in some cases. I haven't yet confirmed if updating Linux kernel (currently using v3.18.8) will fix the issue (rather hard to reproduce), but apparently the problem may be related to incorrect use of fs.fsync(). Related link: https://www.thomas-krenn.com/en/wiki/Ext4_Filesystem#Data_Loss_with_Applications_that_do_not_use_fsync.28.29_correctly

[louischatriot]

Thanks for pointing this out, it seems that ext4 comes with quite a few glitches. That said it also seems like somthing the Node team could do something about. I cannot test here but please keep me updated if you send an accepted bug report to the Node team about this.

[louischatriot]

After diving into the issue I agree that this was actually an issue. As of v1.4.1, NeDB will always force OS to physically write data to disk whenever a compaction or database load happens, so that you can never lose the whole database in case of a machine crash. It doesn't flush on every append for performance reasons, but potential data loss in that case is very limited, a few docs at most. That's what major databases do.

Readme: https://github.com/louischatriot/nedb#persistence

Note that fsync doesn't work on Windows. Not sure whether that's an issue with my machine or you simply can't fsync on Windows. I raised the question with the Node team but any help appreciated!

[zevero]

I am happy that this seems to be resolved. However ... its strange. Yesterday I installed v1.4.1 via npm but today (for my other project) only v1.4.0 is available on npm

[louischatriot]

Indeed, I unpublished v1.4.1 due to cross-platform issues. Nothing major but I prefer to pull this version from npm and releasing when I'm done with the final version. Should be a couple days, I'm having it reviewed by a file system specialist (which I'm not !).

[louischatriot]

I just published v1.5 that fixes the above mentioned issue. It ensures that even if the machine reboots during a compaction, the data is not lost. You can read this part of the readme for more information : https://github.com/louischatriot/nedb#persistence