Calibre Metadata Scan only finds 1 book (Kobo Clara HD)

[cfc-librarian]

• KOReader version:
• Device: Kobo Clara HD

Issue

Have completely switched over to koreader from kobo/nickel and am enjoying the software very much.

One issue is with the search calibre metadata functionality (i.e. browse series), despite deleting and having calibre recreate the driveinfo.calibre and metadata.calibre files. The metadata in calibre exists and is in good order.

When I use the calibre->search settings -> manage libraries -> rescan disk for calibre libraries option, the resulting message says, "Found 1 calibre library with 1 books: 1: /mnt/onboard

The actual calibre library (and the device) holds 3261 books.

crash.log (if applicable)

The following crash log is of an actual crash, but repeating the attempt to scan the calibre libraries does not always generate the crash.

crash.log

04/17/22-14:59:12 WARN  cache is older than metadata, ignoring it
04/17/22-14:59:38 INFO  calibre info loaded from disk (search) in 26244.237 milliseconds: 1 books
04/17/22-14:59:39 INFO  metadata: 1 books imported from calibre in 28796.470 milliseconds
./luajit: not enough memory
!!!!
Uh oh, something went awry... (Crash n°1: 04/17/22 @ 14:59:43)
Running FW 4.30.18838 on Linux 4.1.15-00089-ga2737fc02713 (#34 SMP PREEMPT Fri Dec 27 17:47:13 CST 2019)
Attempting to restart KOReader . . .
!!!!

Thanks for any insight into this issue.

[pazos]

repeating the attempt to scan the calibre libraries does not always generate the crash.

The issue doesn't happen with the recursive search for libraries. It happens parsing the contents of a library.

The entiremetadata.calibre doesn't fit into the memory of your device. Could you tell us its size?. In theory, if it is more than 30MB it should switch to a manual parser, which is not very reliable but at least it won't run out of memory.

In any case you can switch to the manual parser by editing this line: https://github.com/koreader/koreader/blob/master/plugins/calibre.koplugin/metadata.lua#L57

Just put a size (in bytes) lower than the size of your file and perform the search again. If you do so then please tell us if the data that you get with it is ok or some stuff is broken.

[pazos]

https://github.com/grafi-tt/lunajson supports incremental parsing, so using it instead of the manual parser will make it 100% reliable without any OOM issues.

Sadly it is much slower than rapidjson.

[cfc-librarian]

Thanks for the quick response!

The metadata.calibre file is 38.2MB.
The RAM for the Clara HD is 512MB.
After editing line 57 of the file from:

local MAX_JSON_FILESIZE = 30 * 1000 * 1000

to

local MAX_JSON_FILESIZE = 5 * 1000 * 1000

There was no change. Same error occurs.

How can I try the https://github.com/grafi-tt/lunajson ? If you can give me some basic instructions, I will give it a try.

[pazos]

Umm. Then the parser seems broken.

How can I try the https://github.com/grafi-tt/lunajson ? If you can give me some basic instructions, I will give it a try.

Isn't really a suggestion for you as the SAX like API is not implemented. I just pointed as a way to finally fix the issue in the future.

But you can try the other way around.

local MAX_JSON_FILESIZE = 50 * 1000 * 1000 and lets see if rapidjson is able to handle the stuff.

Just to be sure: you need to restart the program after editing the file.

[cfc-librarian]

Success! For anyone else who comes across this situation, the workaround which worked to enable the Calibre Metadata Search was changing this line in
https://github.com/koreader/koreader/blob/master/plugins/calibre.koplugin/metadata.lua

from:

local MAX_JSON_FILESIZE = 30 * 1000 * 1000

to

local MAX_JSON_FILESIZE = 40 * 1000 * 1000

Hope this will be helpful to others.

PS This is obviously a "quick and dirty" workaround. Thank you Pazos for your help with this. All I can say is, "koreader.rocks".

[NiLuJe]

This works because your device happens to have enough RAM to deal with this file w/ rapidjson ;o).

[Frenzie]

That's hardly unusual though, at least on Kobo devices with 512 MB RAM, give or take however filled the cache is at any given moment. :-)

[NiLuJe]

Yup, just throwing it out there for poor Kindle users, where this would blow up in fun and hilarious ways ;o).

[pazos]

@NiLuJe @Frenzie: I'm going to give it a try with https://github.com/grafi-tt/lunajson and replace the whole parser.lua file with a sane implementation.

Since we already have two json modules and there's no reason to add a third one I'm going to add it directly into the plugin, unless you see a reason to make it available from elsewhere.

[Frenzie]

What concretely is the reason to use that one instead of one of the others?

[pazos]

it makes possible to write a set of lua functions and pass them as callbacks that will be executed while traversing the json object.

The other ones require to have the entire json object in memory (to be more precise they will traverse the whole json object making a big lua table).

The SAX like API is really useful if:

a - we're dealing with huge json files
b - we only care about tiny bits inside those files
c - the RAM available is finite.

At the end of the day we're going to use the new module just like we use rapidjson now.

• For little files (up to 30MB) it is faster to parse the whole file and just discard the bits we don't need.
• For the rest of files we just generate a table without unneeded bits

[Frenzie]

Alright, sounds fine to me as a plugin-specific lib.