Sync error : database corrupted

[ghost]

Hi, first, thank you for your work.

I manage to get anki-sync-server to run on my personnal server, I use it with 2 devices (1 one computer, and one phone). But each time that I do make a change on a device or on the other I get this error telling me that my decks can't be merged !

This is a pain, can you do something ?
Thanks again

[slohse]

exact same issue here. Also one desktop computer (Ubuntu 16.04) and one phone (Android app running in the Android runtime of Sailfish OS)

[cdpm]

@slohse: The Ubuntu repositories seem to be stuck at Anki version 2.0.32. Did you install Anki from the package repositories? If so, does the error occur if you manually download and install the current version (2.0.36) from http://ankisrs.net/?

[ninja33]

same issue, always be asked to make full sync even just studied several cards.
btw, I cloned most updated 2.0.36 code dae/anki and replaced 2.0.33 bundled with this release. it seems same problem.

[slohse]

@cdpm yes I used the version from the Ubuntu repositories, but installing version 2.0.36 from ankisrc sadly didn't change anything.

[kiuuik]

same problem with two instances of Anki: "Anki 2.0.36 on Windows8.1" and "AnkiDroid 2.6.0 on Android 6"

[ndl]

I experienced similar problems after installation and after some analysis looks like there are at least two different issues in the current code base that might result in this behavior:

1. Collections objects are created on server without "server=True" flag, therefore USNs returned by the collections objects are incorrect.
2. Collections are not reloaded after full upload to the server, therefore their SCM field is not updated, which results in bogus schema differences detected on every sync attempt.

Both of these issues should be solved with the attached patches (NOTE: the patches are both for anki-sync-server code and anki library that it depends on). However, it seems plausible there might be yet other issues there that I've not yet discovered.

anki-sync-server.patch.txt
anki.patch.txt

[schemacs]

I have also found the usn difference between client and server at sanityCheck2 at anki/sync.py when conflict happens. The patch from @ndl works perfectly, thanks.

[kiuuik]

ndl's patch doesn't seem to work on Windows (but seems to work on Linux,MacOS,Android)

Before modifying my Raspi I decided to test ndl's patch of AnkiSyncServer (from pip, version 2.0.6) in a virtual machine running Debian8.

-Sync from my Debian8 Host with Anki 2.0.31: works

-Sync from another Linux virtual machine with Anki 2.0.36: works

-Sync from macOS 10.11 with Anki 2.0.36: works

-AnkiDroid 2.7: After starting the sync it synced for a couple of minutes. Then it stopped with an error message. I tried a couple of times and prevented my device from sleeping. Finally it went through in about 10 minutes (I have about 10 000 cards and 5000 media files/700mb). Now it seems to work. The regular sync runs quite quickly.

-Sync from Windows 10 Pro "Anniversary Update" with Anki 2.0.36 fails: Before applying ndl's patch it worked as good/as bad as from other systems. I tested two different windows machines. Now on both machines I get the same error message:

Syncing failed:
Traceback (most recent call last):
  File "aqt\sync.py", line 324, in run
  File "aqt\sync.py", line 372, in _sync
  File "aqt\sync.py", line 406, in _fullSync
  File "anki\sync.py", line 691, in download
  File "anki\db.py", line 76, in scalar
  File "anki\db.py", line 43, in execute
DatabaseError: file is encrypted or is not a database

I ran the AnkiSyncServer in debug mode: For the IP of Windows Machine I only found these entries:

192.168.188.21 - - [19/Nov/2016:00:38:27 +0200] "POST /sync/hostKey HTTP/1.1" 200 43 "-" "Python-httplib2/0.7.7 (gzip)"
INFO:root:CollectionThread[/home/kiuuik/AnkiSyncServerData/collections/kiuuik/collection.anki2]: Running meta(*[], **{})
192.168.188.21 - - [19/Nov/2016:00:38:27 +0200] "POST /sync/meta HTTP/1.1" 200 113 "-" "Python-httplib2/0.7.7 (gzip)"
INFO:root:CollectionThread[/home/kiuuik/AnkiSyncServerData/collections/kiuuik/collection.anki2]: Running operation_download(*[<AnkiServer.apps.sync_app.SyncUserSession object at 0x7ff3dc1ebe50>], **{})
192.168.188.21 - - [19/Nov/2016:00:38:31 +0200] "POST /sync/download HTTP/1.1" 200 55090176 "-" "Python-httplib2/0.7.7 (gzip)"

I ran a test with another deck with half the cards/media one day later. On my Windows machine I get the same error message. The output of AnkiSyncServer looks like this:

192.168.188.21 - - [19/Nov/2016:21:40:48 +0200] "POST /sync/hostKey HTTP/1.1" 200 43 "-" "Python-httplib2/0.7.7 (gzip)"
INFO:root:CollectionThread[/home/kiuuik/AnkiSyncServerData/collections/kiuuik/collection.anki2]: Running meta(*[], **{})
192.168.188.21 - - [19/Nov/2016:21:40:48 +0200] "POST /sync/meta HTTP/1.1" 200 112 "-" "Python-httplib2/0.7.7 (gzip)"
INFO:root:CollectionThread[/home/kiuuik/AnkiSyncServerData/collections/kiuuik/collection.anki2]: Running operation_download(*[<AnkiServer.apps.sync_app.SyncUserSession object at 0x7ff5dc5a3290>], **{})
192.168.188.21 - - [19/Nov/2016:21:40:48 +0200] "POST /sync/download HTTP/1.1" 200 3715072 "-" "Python-httplib2/0.7.7 (gzip)"
INFO:root:Monitor is closing collection on inactive CollectionThread[/home/kiuuik/AnkiSyncServerData/collections/kiuuik/collection.anki2]
INFO:root:CollectionThread[/home/kiuuik/AnkiSyncServerData/collections/kiuuik/collection.anki2]: Running _close(*[], **{})

I also tested with Vista and Anki 2.0.32 - in Windows I got the same error message.

[colonelsmoothie]

Hello,

I'm sorry if this sounds like a dumb question, but how do I apply @ndl's patches? I'm running Ubuntu 16.04 and having the same problem requiring a full sync each time.

Thanks!

[colonelsmoothie]

whelp, here's what I tried,

I went into my AnkiServer directory and modified the sync_app.py and collection.py files. I also went into the anki-bundled/anki directory to change the collection.py and media.py files. I did this by adding lines where there was a '+' and removing lines where there was a '-' indicated in @ndl's txt files.

I restarted the server and tried to sync Anki, but got the following error:
Syncing failed:
AnkiWeb encountered an error. Please try again in a few minutes, and if the problem persists, please file a bug report.

Is there something else I need to do?

[slohse]

What you did sounds right in principle, but you should probably have a look at the program patch, that'll make your day easier :)

[colonelsmoothie]

Alright, I did the following

• reset everything to the previous working state by undoing all the changes
• stopped the server

Then I downloaded the patches to my /home/ubuntu/ directory

cd /usr/local/lib/python2.7/dist-packages/AnkiServer
sudo patch -p1 < /home/ubuntu/anki-sync-server.patch.txt

cd /usr/local/anki-bundled/anki
sudo patch -p1 < /home/ubuntu/anki.patch.txt

• restarted server
• restarted Anki

Same error:
Syncing failed:
AnkiWeb encountered an error. Please try again in a few minutes, and if the problem persists, please file a bug report.

Reversing the patches allows Anki to sync, but leaving me where I started with the database corrupted problem.

Anything else I can try?

Thanks for the help, everyone

[colonelsmoothie]

Alright, I think I got it working, I left out one more directory:

cd /usr/share/anki/anki
sudo patch -p1 < /home/ubuntu/anki.patch.txt

Restarted the server and Anki, it's syncing now locally, and works with my Ankidroid. I'll do some further checks when I get back home.

[aikalant]

It looks like I was able to replicate kiuuik's error

Using AnkiServer (with 2.0.36 anki-bundled) with the patch.

Using Android Anki 2.8 everything works fine - downloading, uploading, syncing.

Using desktop Anki 2.0.36, here is the error scenario:

Clean server, start client, upload collection to server, it only uploads the cards, no media, but no error.
Without any changes, click sync again, it asks for one way download/upload, again, clicked upload, this time media is uploaded as well, no errors. Every time from there on out, even with no changes on client, the sync button prompts for one way. Can click upload every time, no issues. If I click download, there is a momentary sync, but from there on out, the entire thing works out. No more one way prompts.

Now, without stopping the server, one way downloads, and new profile downloads and syncing works fine for a while.

The problem seems to happen only when stopping and restarting the server or waiting just a few minutes.

Once that happens, syncing works fine, but on one way downloads and new profile downloads, I get the "DatabaseError: file is encrypted or is not a database" error on the client. Only if I do a one way upload again can I start one way downloading again - until the server is restarted.

Error seems to happen when the client can't verify the integrity of the server's file - from line 689 in /anki/sync.py

    # check the received file is ok
    d = DB(tpath)
    assert d.scalar("pragma integrity_check") == "ok"
    d.close()
    # overwrite existing collection