"Updating note database" appears to be stuck

[jmanuel1]

I have a lot of notes, most of which were imported from Evernote as HTML. When Jarvis starts to update the note database, it gets to a certain number of notes, usually 250, before not progressing any further. I haven't noticed any logs in DevTools from Jarvis, and the "Universal Sentence Encoder" file exists and has some data in it. And occasionally the related notes search comes up and I can actually use it, but not often.

I wonder if one of my notes is causing the issue, but since I didn't find logs, I wouldn't know which one.

[alondmnt]

I'm sorry to hear that @jmanuel1. Never happened to me in the current version of the plugin, but this doesn't mean that there is no bug there. It's certainly possible that one particular note breaks the code for some reason.

It's going to be a little tricky to debug without an example note that enables reproduction of the bug. And because the plugin processes multiple notes in parallel (so not even necessarily the 250th in line) it's going to be tricky to figure out which note fails (unless we find a good error message). It also occurred to me that I personally don't use HTML notes, and it's generally a good idea to do some testing around them (will work on it).

• Have you enabled debugging in the app before checking the console?
• Since there is no error message from Jarvis, is it possible that you have a huge note that may take a very long time to process compared to all other notes? (I can already tell that your notes tend to be larger than mine on average, based on the size of the DB.)
• If you have notes or notebooks that you suspect that may cause the issue, you can place them in a notebook (or sub-notebook), and mark it with Tools-->Jarvis-->Exclude notebook from note DB. It's reversible, anyway.

[jmanuel1]

I haven't enabled debugging. I also haven't checked the size of my notes... I'll get back to you on both of these.

[jmanuel1]

Here's what logs I managed to capture that seemed relevant

from devtools:
all-logs.txt
jarvis-logs.txt

Unfortunately, I couldn't just right-click and select "save as" in the devtools console, so I copied the above logs manually. They're a bit soupy.

from profile directory:
profile-folder-logs.txt

@alondmnt

[jmanuel1]

I've been moving my largest notes to a folder I made specifically to exclude them from the database, and now the update stops at 450 notes.

EDIT: Now it's 750.

[alondmnt]

Hi @jmanuel1, sorry for the long wait (I was preoccupied with... life). I went over the logs, and although I could not recognize any explicit error, I think that the dynamics of the DB construction are visible in the logs. It's easy to get from the logs the list of the last processed batch of notes, in order to identify the culprit note(s) that stop the update from completing.

Here's how to read it:

• Every page of notes starts with a message like this one: .... Got message (3): joplin.data.get ['notes'] {fields: Array(5), page: 1, limit: 50} (get the next 50 notes)
• Then you'll see 50 messages like this one when the processing of each note starts: .... Got message (3): joplin.data.get (3) ['notes', 'd2e4331630a146be9bfc070704b5d6e2', 'tags'] {fields: Array(1)}

So the last group of 'tags' messages contains all the notes that were being processed while Jarvis failed for some reason. 'd2e4331630a146be9bfc070704b5d6e2' in this case is the note ID, and you can use it to locate the note and exclude it from the Jarvis DB, or share it if you wish so I can try to figure out what went wrong. I understand that 50 may be a little too ambiguous, so I'm attaching a custom Jarvis version that processes notes one by one (this is the only change from the current v0.7.0).

Once you've identified a note that failed, you may add the exclude.from.jarvis tag to it, and the next time the update runs it will be ignored. You may repeat this as many times as necessary.

Hope this helps.

Unzip this file and manually install the plugin: joplin.plugin.alondmnt.jarvis.jpl.zip

[jmanuel1]

@alondmnt Sorry for letting this issue drop by the wayside. I do have something of note, though: when I installed the custom Jarvis version that processes notes one at a time, it seems like the process started to finish! I haven't actually seen it finish since it takes a while, but at the very least, the database update gets very far.

I might try to un-ignore some notes to see if the update process gets stuck on any of them, and also record the process to get evidence that it completes.

[alondmnt]

Thanks for the update @jmanuel1.
Note that if you use the latest Jarvis v0.8.2 there's a new setting, Notes: Parallel jobs, where you can select the number of notes that will be processed concurrently during a database update. Setting it to 1 is the same as the custom version that I posted above.

[jmanuel1]

I installed the latest Jarvis and didn't change the Parallel jobs setting (it's 10), and Jarvis completed updating the database. So, it seems like my issue's fixed. Thanks for the help.