Quickroam

Is your org-roam slow?

If you’re like me, you think org-roam has the right idea, but the current implementation is not reliable or fast enough for day-to-day work.

1. The end-user commands take 5 seconds to get the minibuffer ready.
   • A workaround by memoizing works well most of the time, but if I create a new node and immediately want to insert it elsewhere, I have to wait for an idle timer to update the memoization.
2. Very large Org files take 10 seconds to save with org-roam-db-autosync-mode.
   • I can turn the mode off, but then I need to manually run M-x org-roam-db-sync every now and then! Not great when I’m in the zone and writing.

As best I can tell by reading the source code, problems #1 and #2 are totally separate. Problem #1 should be fixable, but #2 arises from the fact that org-roam is very generalized – for example, it lets you plug in user-defined functions to exclude or include a node. That’s nice and hackable, but it means that org-roam is forced to use Org functions to “properly” parse every file it scans, incurring all the penalties of doing so.

Fix

This package provides alternatives to common commands:

• org-roam-node-find – use quickroam-find
• org-roam-node-insert – use quickroam-insert

They don’t look up the org-roam DB at all, just rely on ripgrep. They’re fast, fixing problem #1, and they always find up-to-date hits, which means you can set org-roam-db-update-on-save to nil to fix problem #2.

If you do the #2 fix, be aware that backlinks won’t update until you run M-x org-roam-db-sync RET.

Setup

Install ripgrep on your computer. Then add to Emacs initfiles something like this, assuming you install packages with straight.el:

(use-package quickroam
  :straight (quickroam :type git :host github :repo "meedstrom/quickroam")
  :defer t)

(add-hook 'org-mode-hook #'quickroam-enable-cache)
(global-set-key (kbd "<f2> f") #'quickroam-find) ;; or whatever key you use
(global-set-key (kbd "<f2> i") #'quickroam-insert)

Future work

(2024-04-23: quickroam has hit version 1.0 and is not expected to change. Now I dogfood a spin-off with more features: org-node!)

This package is a proof-of-concept that ripgrep can collect part of the data needed by the org-roam DB. However, simple regexps cannot collect all that’s needed. Some challenges:

• TODO/DONE keywords
  • Need awareness of buffer-local #+seq_todo settings
• The outline path
  • What it’s used for: https://fosstodon.org/@nickanderson/112249581810196258
• Backlink context: the name of the subtree-node in which a link is found
  • Pretty important since backlinks butter our bread, right?
  • If we mandated the rule of “one file, one node” like zk/orgrr/denote do, we’d only need to grab the file title. But that’s not why we’re here. Org-roam’s nested nodes are the best thing since sliced bread, but they need us to parse the Org syntax.
    • One option is to satisfice with a poor-man’s backlinks buffer by letting it just show, say, 2 lines above and below each link, reminiscent of a diff output.
    • Another option is to cache the backlinks as a property or drawer(!) like org-super-links does.

You can see all the data org-roam collects in org-roam-db–table-schemata.

Three ways forward:

1. Don’t parse Org syntax, and compromise on the kinds of data to collect.
2. Parse Org syntax, but do it faster.
   • Maybe rewrite org-roam-db-update-file to scan the buffer with org-ql? Don’t know if it’d be faster.
3. Parse Org syntax ahead-of-time.
   • View the SQLite database as a first-order cache, and create a second-order cache consisting of the org-super-links approach mentioned earlier.
   • It’s a powerful concept! Record directly into property drawers everything that’s impossible to write a regexp for, so you can later just grep the values.
     • Now syncing the DB has been reduced to a job for grep. No lines of Org code need to run.
       • We could even get rid of the DB, since ripgrep-based “queries” would be fast enough.
         • But, SQL queries are a handy tool for package devs, so either we keep the DB or we make sure that all the same things can be done in org-ql. In fact, alphapapa has expressed interest in giving org-ql a SQL backend, so it looks like a promising place for some synergy to happen!
         • Anyway, it costs us nothing to keep the DB. If worried about stale data, we can just rebuild it much more often.
     • Caveat: we must wrap many user commands. Let’s say you record the subtree’s outline path as a property :CACHED_OLPATH:, and every time you indent or outdent a subtree, some advice automatically updates this property.
       • In real life, we can’t trust that 100% of the time, so it’d be good to have a command to validate all the properties in all files. It’s still a win because you don’t need to run such code on every (org-roam-db-sync 'force), you’d just run it as a kind of linter once a month!
       • If you wanted to cache a ton of data, you could compress it by gzip and put the base64-encoded string on a single property :ROAM_METADATA: or some such.
   • An alternative, I guess may work too: keep a table in the SQLite database for every subtree, and avoid using org-roam-clear-file so often.

Thanks

Inspired by a Mastodon conversation with the orgrr author :)