Async API?

[Al2Klimov]

Hello @JoshuaWise,

At the moment I'm looking for a SQLite3 Node.js lib and have also seen yours.
You claim that your "Easy-to-use synchronous API" is "faster than an asynchronous API".

I have no idea what your test env was, but it definetly didn't contain any external HDD for this. At least not one which falls asleep after not being used some time and just before the start of your benchmark.
Such devs are likely to spend a couple of seconds to wake up and another couple of ms to actually serve your I/O request.

During these couple of seconds + couple of ms an asynchronous lib could let the event loop do other things (maybe even just cleaning up all the things on SIGTERM to leave /var/lib/w/e in a consistent state).

The same (event loop) thing applies to a SELECT FROM an e.g. 0.5 to 1TB sized SQLite DB w/ a large result set.

Long story short: Yes, developers are lazy and grasping at straws to get their things done, but couldn't you add an optional async API for people... like me (who care about race conditions).
I could also do that, but before I write a single LOC, I'd like to know whether this is even desired.

Best,
AK

PS: An async API could be encapsulated w/ Promises (one per method) and these could be used w/ the bleeding edge async/await (Hello babel.js!) just as easy as things are right now.

[bobOnGitHub]

Just a heads-up there AK ; I tried async/await/(native)promises in an effort to avoid callback hell with db updates ( using node-sqlite3 ) and hit massive performance issues ; 10+ times slower than callbacks ( similar to this guy ; https://www.npmjs.com/package/zco ) .

[Al2Klimov]

1. Developing time is more expensive than el. power, bandwidth or datacenter leasing
2. Code is (usually) written once and then read again by s/b else and changed
3. :

(...) An async API could be encapsulated w/ Promises (...)

It doesn't have to be.

[JoshuaWise]

This is a duplicate of #32.

Long story short...

Point 1:

If you're using a slow hard drive that's constantly falling asleep, use npm/sqlite. You've already chosen to prioritize machine cost over development cost, so you should be fine with that. If you want to prioritize development cost instead, choose a very affordable solid-state drive (digitalocean, for example) and this problem is eliminated.

Point 2:

You might still be concerned about long reads/writes blocking other things...

- Long writes:

Write transactions should be isolated to a single processes, since SQLite can only have one concurrent writer on a database anyways. Having multiple writers is bad design because it introduces locking and polling penalties. So just don't do that—use SQLite properly. If you do, this problem is eliminated.

- Long reads:

Long read transactions are a valid concern, because SQLite can have concurrent readers. In most cases, the overhead of context switching and locking mechanisms (required for an async API) is much more than the overhead of blocking while reading some data even with cache disabled. The exception, of course, is when reading very large amounts of data. In these cases, npm/sqlite may be better for you. Because that library has many problems (234 open issues), it's recommended to use better-sqlite3 for as much as possible, and only use npm/sqlite for the occasional very large read operation.

Point 3:

Because async read operations can (although, they don't always do) have an advantage over blocking read operations, it might seem like a great idea to add an optional async API to better-sqlite3, to avoid using npm/sqlite altogether. However, I disagree with this for two reasons:

1. It's impossible to implement custom SQL functions with an async API, because Node.js is single-threaded, without exception.
2. It expands the scope of better-sqlite3 to do something that is done relatively infrequently, which is something that npm/sqlite can already do for you. In order to keep modules healthy it's important to follow the "do only one thing, and do it very well" philosophy, and fight feature creep.

Conclusion

I hope this provides a sufficient explanation for why an async API will not be added to better-sqlite3.

• An async API would be incompatible with existing features like custom SQL functions.
• If the system is designed correctly, it is rarely an issue.
• When it is an issue, you can just use npm/sqlite.

[Al2Klimov]

Fine. So I'll use your lib's arch enemy.

Thank you and goodbye.