Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
As a prelude to API syncing, in a private branch I've been working on switching the data layer to use asynchronous database access, which is necessary to avoid freezing the UI. (Zotero predates asynchronous database access in Mozilla, but synchronous access on the main thread is now strongly discouraged because of its effect on UI responsiveness.) Unfortunately, due to conflicts between synchronous and asynchronous access, it looks like we'll need to switch over all DB queries in Zotero to be async. More unfortunately, async DB access means returning promises from many core data layer methods, which means rewriting essentially the entire codebase to use promises. This seems like a terrible idea that we'll regret later, but we also haven't come up with a better solution.
This will have fairly major effects on the Zotero client ecosystem, so I'm creating this ticket to share some of the issues involved. I'll be pushing my private
To do a more-or-less straight conversion of the existing JS API, we'd have to make nearly everything return a promise. This is what I started off doing, but it would result in a much wider spread of asynchronicity to other parts of the code, with generators and yields and promises everywhere. For example, anything that needed to know if an item was a note would have to become asynchronous, because
In addition to creating much more overhead and requiring many more changes, this spread of asynchronicity would also mean that there wouldn't be a good way to predict whether a given method was async due to some internal dependency on some other async method, which I think would make it very hard to make changes down the line, since some small new bit of functionality might require making an async call from a previously synchronous method.
We can cut down on the number of promises by adjusting where database access happens. So for the above example, we can just say that any
Unfortunately that still leaves a lot of things that are currently lazily loaded. For example, item field data is currently not loaded until there's a
With promises and generators, that would turn into this:
However, that would mean that anything that needed any non-primary item field would have to be asynchronous as well, which would have pretty far-reaching effects.
Unfortunately, the lazy-loading is pretty important for performance. Right now, when first opening Zotero, we load all data necessary to display and sort the middle pane, and for large libraries that can already result in a notable delay. If we loaded all of an item's field data whenever generating a
My solution has been to trade some elegance for backwards-compatibility and efficiency, with new public
This results in many fewer promises/yields and allows more existing code to remain unchanged. The synchronous methods also only fail if their cached values don't exist, which allows for more granular loading in some situations — for example, right now we load entire columns of field data for the middle pane and stuff them into the objects without triggering full loads.
Finally, be aware that I will likely rebase this branch before merging it into master (in order to keep a sane commit history, avoid breaking git bisects, and keep upgrade steps as efficient as possible), so if you base any work on it, be prepared to handle the rewind. Apologies in advance.
added a commit
Aug 7, 2014
The bulk of promisification work is now complete (with a few big issues remaining, plus a few minor things that aren't on that list), so I'm renaming the
added a commit
Mar 7, 2016
@dstillman I might have relevant experience for the kind of thing described in daf4a8f ... I've just ran few quick tests, and it looks like I can load/save a 3k (OK, OK, 2885) item database in approx 95/100 ms respectively on average. I can't say that the constraints I'm willing to work under fit your case though.