[Feature Request] Sync Progress with KOReader

[raygan]

Is your feature request related to a problem? Please describe.
I see Kobo progress sync and Hardcover.app are being worked on. I think it would also be great to sync reading progress with other ereaders running KOReader. This would include jailbroken Kindles and ereaders from Pocketbook etc, as well as Android devices.

Describe the solution you'd like
Calibre-web-automated could act as a KOReader sync server, adapting or including the project here: https://github.com/koreader/koreader-sync-server

For books in the Calibre library, progress could be displayed on the book page and synced to Hardcover.app.

Additional context

Komga, a comics and manga focused self-hosted library app, added a similar integration recently.

The KOReader software can currently only be configured with a single syncing server. This means that a project like Calibre-web-automated would either need to handle syncing of progress data for any book, not just those found in its database, or it would need to proxy those sync operations to a different sync server. Komga currently does neither, but has suggested it may add the option to proxy syncs for books it doesn't recognize in the future.

[BredMan0]

This would be great to have.

[5rxiq]

+1 would love to see this

[Tom-Enns]

+1 I would also love to see this.

[DoubleNaught7]

I would love to see this as well. I much prefer CWA to Komga overall, but the progress sync was enough to make me spin up an instance to host any books that I'm currently reading in KOReader. I'd love to be able to do that directly in CWA.

[BredMan0]

Indeed, the lack of KOReader sync functionally is a dealbreaker.

[sirwolfgang]

The syncing between Koreaders is easy enough; it's just a key-value store, but the issue is that the key is a digest calculated on the device, which makes it basically encrypted.
https://github.com/koreader/koreader/blob/8db0dee5e07343bfb9af845d69a73734366dc5dc/plugins/kosync.koplugin/main.lua#L600-L606

The solution would be to calculate the two different digests on the CWA's side, then check the store for matches, and finally pull those in. It could work, but it will be fragile to any changes to Koreader's implementation. Additionally, you'd need to calculate the digests for each file/version.

The more resilient option would be to fork the KOsync plugin, and tweak it to send identifiers matching what CWA has; but it would also mean that everyone needs to install a custom plugin. As a standalone feature, not sure the juice is worth the squeeze here, but if CWA starts to integrate with Hardcover, having this all sync together seems choice.

[sirwolfgang]

#387

[sirwolfgang]

The initial part of this has been implemented in #425

The next step is to reimplement KOReader's MD5 algo and then use it to start harvesting the syncing data into CWA.

[alva-seal]

If this gets implemented it would be great to also sync the progress to hardcover as it is planed for kobo users

[Jojont54]

I think #461 is also relative to this request.
Implementing progress in CWA allows continuous reading through webreader.

MD5 Algo in python: (in lua "<< -1" goes to 0, in python it's forbidden)

def partial_md5(filepath):
    import hashlib
    step, size = 1024, 1024
    md5_hash = hashlib.md5()
    try:
        with open(filepath, 'rb') as file:
            for i in range(-1, 11):
                offset = 0 if i == -1 else step << (2 * i)
                file.seek(offset)
                sample = file.read(size)
                if not sample:
                    break
                md5_hash.update(sample)
    except Exception:
        return None
    return md5_hash.hexdigest()

It calculates md5 1024 bytes from 0, then 1024 bytes from 1024, then 1024 bytes from 4096 etc.
The main difficulty is to store this data in relation to books on the database.

[alva-seal]

I was thinking about it:

My idea for storing is to have a „translation“ table in the database that relates the md5 to a book id.
This way a book also can have multiple hashes attached eg after changed metadata are written to the file which obviously would change the hash. And if someone wants to know the progress of a book the newest dataset of the assigned hashes is choosen.

[alva-seal]

Kobo progress location looks at a first glance very similar to koprogess so I would be hopeful we could even achieve to have progress sync between kobo and koreader

[Jojont54]

Another difficulty I found here is the version stored on the server can be changed with WebUI download button or OPDS download,
But not always.

Refer to my comment here : #476

That will break the kosync.

[Jojont54]

Also relative to :
#483

Embed metadata on calibre gui realign stored file with metadata, then Download / OPDS process get the same hash.

https://github.com/kovidgoyal/calibre/blob/master/src/calibre/db/cli/cmd_embed_metadata.py
Calibre CLI usage.

The computation of the Hash should then be redo after a metadata integration to align all version of the epub.