Concurrent writes

[rooftopsparrow]

Unfortunately, nosqlite will jumble output if called rapidly, as implied by your checklist item "Buffer writes internally", and shown in some tests I've written.

I'm needing buffered write fairly soon for my own project, and was wondering what your ideas are for this, and if it is doable in my timeframe, I can take on that challenge.

But I'm not an 'expert' on fs.writeStream or other methods, so I would love to hear your ideas.

Thanks!

• rooftopsparrow

[pksunkara]

Sorry, was a bit busy. Could you show me your tests in a gist? I think I meant two things when I meant "Buffer writes".

[rooftopsparrow]

Take a look at this gist.

This is a super simple case, but illustrates the problem if this happened in a web server or other system( i.e same request by different people at the same time). Notice the test.json file is incorrect json: some of the first object bled over the second object.

The issue is the event loop times out on the first write, then the second one starts and writes to the same file, over writing it, then back to the first at where it left off, etc... The first to finish renames the file, with some bad junk in it, then when the second one tries to rename, it fails.

My first attempt at fixing this just uses the cuid library, which we included for the auto id generation, as the temp name for each _write. That way every "event" would have its own file to work with. But I do not know if this is safe enough.

If you had insight into this, I would love to hear your thoughts.

[pksunkara]

The example you have shown can be solved by using db.postSync(obj);. Every api has a synchronous implementation too. But I get what you are saying. The initial plan was to use temporary files and use their timestamps.

[rooftopsparrow]

Yes, I wanted to avoid using db.postSync, but it will most likely work for my current application. However, I would still like to look at this for purely academic reasons. 📚

[pksunkara]

Yes. I had this idea of using temporary randomly named files and use their timestamps when merging together.

[rooftopsparrow]

How would you know when the timestamped file was completely written? For example, the first file is writing, but doesn't finish, a second file writing but finishes, and now wants to complete. How would it know that the earlier file has finished writing so it can apply that first, then the newer one second?

Perhaps I'm thinking about this too much, but I just find this stuff interesting. 😄