Database startup error: Error: SQLITE_NOTADB: file is not a database

[eppfel]

• I have searched open and closed issues for duplicates
  Related, but outdated: Signal won't show on screen after launch on Windows 10 x64 #2712 Signal won't start after upgrade to 1.15.0 #2609

Bug Description

I just started Signal and receive the error Database startup error: Error: SQLITE_NOTADB: file is not a database with the option to delete all data and start new.

What is strange is, that I have a Signal and Signal-betafolder in ~/Library/Application Support/. The files in beta however are all created on 2020-01-08 and no changes were made after that. I thought I was using beta but somehow I did not.

Also interesting that it tries to migrate the data with each restart.

I have used Signal on and off on that machine, not on a daily basis.

Screenshots

Platform Info

Signal Version:

1.35.2

Operating System:

OS X 10.14

Linked Device Version:

4.71.2

Link to Debug Log

No link, as App did not start, so here the log file.

{"name":"log","hostname":"xBook.home","pid":16348,"level":30,"msg":"app ready","time":"2020-09-12T07:47:54.231Z","v":0}
{"name":"log","hostname":"xBook.home","pid":16348,"level":30,"msg":"starting version 1.35.2","time":"2020-09-12T07:47:54.231Z","v":0}
{"name":"log","hostname":"xBook.home","pid":16348,"level":30,"msg":"migrateDatabase: Migration without cipher change failed","time":"2020-09-12T07:47:54.261Z","v":0}
{"name":"log","hostname":"xBook.home","pid":16348,"level":30,"msg":"Database startup error: Error: SQLITE_NOTADB: file is not a database","time":"2020-09-12T07:47:54.263Z","v":0}
{"name":"log","hostname":"xBook.home","pid":16348,"level":30,"msg":"sql.initialize was unsuccessful; returning early","time":"2020-09-12T07:48:24.204Z","v":0}
{"name":"log","hostname":"xBook.home","pid":16810,"level":30,"msg":"app ready","time":"2020-09-12T08:45:16.683Z","v":0}
{"name":"log","hostname":"xBook.home","pid":16810,"level":30,"msg":"starting version 1.35.2","time":"2020-09-12T08:45:16.684Z","v":0}
{"name":"log","hostname":"xBook.home","pid":16810,"level":30,"msg":"migrateDatabase: Migration without cipher change failed","time":"2020-09-12T08:45:16.698Z","v":0}
{"name":"log","hostname":"xBook.home","pid":16810,"level":30,"msg":"Database startup error: Error: SQLITE_NOTADB: file is not a database","time":"2020-09-12T08:45:16.700Z","v":0}
{"name":"log","hostname":"xBook.home","pid":16810,"level":30,"msg":"sql.initialize was unsuccessful; returning early","time":"2020-09-12T08:45:21.291Z","v":0}

the config.js holds a key.

[scottnonnenberg-signal]

@eppfel When you see this error, it means that something has gone wrong with your database on disk. It happens most frequently when your computer loses power unexpectedly, or you have to shutdown your computer abruptly. Did anything like that happen recently?

[eppfel]

Thanks for getting back to me. I don't recall it, but the laptop is old so the battery discharges quickly.

[scottnonnenberg-signal]

Prior discussion about this error is here: #2609

[eppfel]

This seems different in the sense, that it fails migrating the databse in my case. Could this be related to an update?

[scottnonnenberg-signal]

It's definitely the same thing. We are prepared to migrate a database (encryption ciphers or database schema, as needed) on every database open.

[bjdooi]

I'm having the same exact issue (probably due to my computer randomly shutting off during an update). I contacted support@signal.org like the previous issues said to do but haven't heard back from them yet

[primeos]

This just happened to me too after a normal reboot (no crash or anything and a FS corruption seems unlikely):

{"name":"log","hostname":"jarvis","pid":15241,"level":30,"msg":"app ready","time":"2020-12-08T14:25:15.449Z","v":0}
{"name":"log","hostname":"jarvis","pid":15241,"level":30,"msg":"starting version 1.38.2","time":"2020-12-08T14:25:15.449Z","v":0}
{"name":"log","hostname":"jarvis","pid":15241,"level":30,"msg":"migrateDatabase: Migration without cipher change failed","time":"2020-12-08T14:25:15.478Z","v":0}
{"name":"log","hostname":"jarvis","pid":15241,"level":30,"msg":"Database startup error: Error: SQLITE_NOTADB: file is not a database","time":"2020-12-08T14:25:15.481Z","v":0}
{"name":"log","hostname":"jarvis","pid":15241,"level":30,"msg":"sql.initialize was unsuccessful; returning early","time":"2020-12-08T14:25:19.715Z","v":0}

I'm on GNU/Linux with Signal-Desktop 1.38.2. I'll try to find out more.

Updates:

This is the log of the last application shutdown:

{"name":"log","hostname":"jarvis","pid":26735,"level":30,"time":"2020-12-07T17:14:51.665Z","msg":"Sending a keepalive message","v":0}
{"name":"log","hostname":"jarvis","pid":26735,"level":30,"time":"2020-12-07T17:15:46.961Z","msg":"Sending a keepalive message","v":0}
{"name":"log","hostname":"jarvis","pid":26735,"level":30,"time":"2020-12-07T17:16:39.017Z","msg":"Removing notification","v":0}
{"name":"log","hostname":"jarvis","pid":26735,"level":30,"time":"2020-12-07T17:16:40.032Z","msg":"SQL channel job 24509 (updateConversations) succeeded in 12ms","v":0}
{"name":"log","hostname":"jarvis","pid":26735,"level":30,"msg":"close event {\"shouldQuit\":false}","time":"2020-12-07T17:16:42.391Z","v":0}
{"name":"log","hostname":"jarvis","pid":26735,"level":30,"msg":"requestShutdown: Requesting close of mainWindow...","time":"2020-12-07T17:16:42.393Z","v":0}
{"name":"log","hostname":"jarvis","pid":26735,"level":30,"time":"2020-12-07T17:16:42.077Z","msg":"Sending a keepalive message","v":0}
{"name":"log","hostname":"jarvis","pid":26735,"level":30,"time":"2020-12-07T17:16:42.396Z","msg":"background/shutdown","v":0}
{"name":"log","hostname":"jarvis","pid":26735,"level":30,"time":"2020-12-07T17:16:42.396Z","msg":"attachment_downloads/stop: disabling","v":0}
{"name":"log","hostname":"jarvis","pid":26735,"level":30,"time":"2020-12-07T17:16:42.396Z","msg":"MessageReceiver: stopProcessing requested","v":0}
{"name":"log","hostname":"jarvis","pid":26735,"level":30,"time":"2020-12-07T17:16:42.396Z","msg":"MessageReceiver.close()","v":0}
{"name":"log","hostname":"jarvis","pid":26735,"level":30,"time":"2020-12-07T17:16:42.396Z","msg":"WebSocketResource.close()","v":0}
{"name":"log","hostname":"jarvis","pid":26735,"level":30,"time":"2020-12-07T17:16:42.396Z","msg":"drained","v":0}
{"name":"log","hostname":"jarvis","pid":26735,"level":30,"time":"2020-12-07T17:16:42.397Z","msg":"MessageReceiver: unregister batchers","v":0}
{"name":"log","hostname":"jarvis","pid":26735,"level":30,"msg":"requestShutdown: Response received","time":"2020-12-07T17:16:42.449Z","v":0}
{"name":"log","hostname":"jarvis","pid":26735,"level":30,"msg":"before-quit event {\"readyForShutdown\":true,\"shouldQuit\":false}","time":"2020-12-07T17:16:42.450Z","v":0}
{"name":"log","hostname":"jarvis","pid":26735,"level":30,"msg":"close event {\"readyForShutdown\":true,\"shouldQuit\":true}","time":"2020-12-07T17:16:42.450Z","v":0}

There seems to be no indication that anything did go wrong. I've created a backup right after closing Signal-Desktop but the DB (~/.config/Signal/sql/db.sqlite) from the backup has the same hash and is also corrupt (I cannot open it manually using sqlcipher and since it's encrypted I cannot really investigate why...).

A comparison of the file sizes (last working backup vs. corrupt DB):

$ du -h Signal/sql/db.sqlite Signal-corrupt-sqlcipher-db/sql/db.sqlite
56M	Signal/sql/db.sqlite
57M	Signal-corrupt-sqlcipher-db/sql/db.sqlite
$ du -b Signal/sql/db.sqlite Signal-corrupt-sqlcipher-db/sql/db.sqlite
58351616	Signal/sql/db.sqlite
59084800	Signal-corrupt-sqlcipher-db/sql/db.sqlite

But only the first 16 bytes of db.sqlite seem to be identical in both files (I didn't look at the SQLCipher documentation; not sure if it should look that way or not).

[ryantrinkle]

I'm trying to upgrade from v1.37.3 to v1.39.4, and I"m getting this same error (NOTADB). If I manually start v1.37.3, everything still works and all data is still present.

[ryantrinkle]

I've now narrowed this down a bit: I was able to step through versions up to 1.38.2, which runs successfully. Running 1.39.2 (which seems to be the next release after 1.38.2) results in NOTADB.

Set Windows Application User Model ID (AUMID) { appUserModelId: 'org.whispersystems.signal-desktop' }                                              
NODE_ENV production
NODE_CONFIG_DIR /nix/store/8x3rqj3j9dg2g3f842xwymrhffvi60y6-signal-desktop-1.39.2/lib/Signal/resources/app.asar/config
NODE_CONFIG {}
ALLOW_CONFIG_MUTATIONS undefined
HOSTNAME undefined
NODE_APP_INSTANCE undefined
SUPPRESS_NO_CONFIG_WARNING undefined
SIGNAL_ENABLE_HTTP undefined
userData: /home/ryan/.config/Signal
config/get: Successfully read user config file
x-attr dependency did not load successfully
config/get: Successfully read ephemeral config file
making app single instance
{"name":"log","hostname":"aji","pid":21418,"level":30,"msg":"app ready","time":"2020-12-24T02:33:44.571Z","v":0}
{"name":"log","hostname":"aji","pid":21418,"level":30,"msg":"starting version 1.39.2","time":"2020-12-24T02:33:44.571Z","v":0}
{"name":"log","hostname":"aji","pid":21418,"level":30,"msg":"media access status undefined undefined","time":"2020-12-24T02:33:44.572Z","v":0}
{"name":"log","hostname":"aji","pid":21418,"level":30,"msg":"migrateDatabase: Migration without cipher change failed","time":"2020-12-24T02:33:44.586Z","v":0}
{"name":"log","hostname":"aji","pid":21418,"level":30,"msg":"Database startup error: Error: SQLITE_NOTADB: file is not a database","time":"2020-12-24T02:33:44.587Z","v":0}
{"name":"log","hostname":"aji","pid":21418,"level":30,"msg":"sql.initialize was unsuccessful; returning early","time":"2020-12-24T02:33:48.441Z","v":0}

[scottnonnenberg-signal]

@ryantrinkle It's really surprising that the changes from v1.37 to v1.39 would result in that kind of error, since no low-level database changes are included. How are you installing Signal Desktop?

[primeos]

I hate Signals way of dealing with long term conversations and history archival, it sucks - there are no proper ways of dealing with this. @moxie0

+1, please do something about this.

Finding and fixing the bug might be a challenge but in any case Signal-Desktop should (IMO) at least:

1. Support making backups and importing them without relying on 3rd party tools: Allow exporting a backup from Signal Desktop #522
   • https://support.signal.org/hc/en-us/articles/360007059752-Backup-and-Restore-Messages ("Manually transferring folders is not supported and may result in errors. Signal uses a stateful protocol, meaning that you cannot simply copy files to another instance of Signal or copy files after linking and using Signal.")
   • 3rd party: Package tools for dealing with signal backups NixOS/nixpkgs#106038
2. Support fetching(/importing) older messages from the phone (if the user wants it): [Feature request] Synchronise all messages from mobile app during desktop app installation #1651

[neonfuz]

I have been getting network errors in signal today so I looked at issues for my distro (nixos) and found this bug, so I quit signal to backup my signal folder, and upon re-opening it I had a corrupted database... very annoying, my phone is broken right now so its going to be a pain getting access to signal now.

[fdietze]

I have the same problem. So what are my options now? Is there a way repair the corrupted database? Do I have to delete my history? Should I wait for a fix?

[backuitist]

You could check the first 16 bytes of your db file:

$ hexdump -C ~/.config/Signal/sql/db.sqlite | head -1

You're supposed to see this:

00000000  53 51 4c 69 74 65 20 66  6f 72 6d 61 74 20 33 00  |SQLite format 3.|

On my broken DB I had this:

00000000  f0 fd 28 28 7e 5b 5a d7  8f 3f d1 ea 60 73 1c 47  |..((~[Z..?..`s.G|

I couldn't find any tool able to repair a DB with a bad header...

[fdietze]

Hmm, I have:

00000000  f7 93 95 e1 ef b4 ad ee  98 3b ac 32 54 15 84 10  |.........;.2T...|

😞

[mehdibo]

Same error just happened to me today

Database startup error:

Error: SQLITE_NOTADB: file is not a database

[fdietze]

Since I wanted to use Signal on my Desktop again, I just deleted my whole chat history. That felt really bad...

[panicfarm]

This Database startup error: Error: SQLITE_NOTADB: file is not a database on Windows 10 makes the program unusable- it has destroyed my chat history on two occasions. It only happens after the app autoupdate, when I quit and restart the same version of Signal, the DB is opened fine.
When I chose "Delete All Data" on the error dialog, it shows

Error: EBUSY: resource busy or locked, unlink 'C:\Users\[REDACTED]\AppData\Roaming\Signal\sql\db.sqlite'
    at Object.unlinkSync (fs.js:930:3)
    at Function.rimrafSync [as sync] ([REDACTED]\app.asar\node_modules\rimraf\rimraf.js:306:17)
    at removeDB ([REDACTED]\app.asar\app\sql.js:1150:10)
    at Object.initialize ([REDACTED]\app.asar\app\sql.js:1123:13)

Using Process Explorer, I could not find a process that was locking db.sqlite

[sonergonul]

Now, this is on Hacker News: https://news.ycombinator.com/item?id=26292299

[justinclift]

@backuitist If Signal is using SQLCipher, as mentioned earlier in this thread, then the header of the encrypted database isn't the standard SQLite header.

For example, this is the header from a SQLCipher database with some generated ~random data for testing purposes:

$ hexdump -C 2.5mb-encrypted-abc123.sqlite | head -1
00000000  ff 3f fc f2 74 1a 61 44  5c c7 44 0e bb 0b a1 42  |.?..t.aD\.D....B|

That database file opens fine with any client that can use SQLCipher version 4.x. eg DB Browser for SQLite

Note, this is the above test database itself (not a Signal one), just in case it's useful for someone to investigate with. Password is "abc123":

• 2.5mb-encrypted-abc123.zip <-- thrown inside a .zip file, so GitHub would accept the upload

[justinclift]

@ryantrinkle

I'm trying to upgrade from v1.37.3 to v1.39.4, and I"m getting this same error (NOTADB). If I manually start v1.37.3, everything still works and all data is still present.

That kind of sounds like the newer version(s) of Signal is using different encryption settings than the older version. Would personally kind of expect the newer version to at least try reading in the "old" database using the "old" settings, and then write out the new database using the "new" settings. Maybe something is going wrong with that process?

@norpol Sounds like the journalists losing their data should revert to the older Signal version until this problem is resolved. Hopefully that's something they can do.

[norpol]

@justinclift Thanks, I'll forward this information. Update: Suddenly just asking to re-link, so the archive is thankfully not gone.

Regarding SQLCipher At least for the my current version (1.39.6) the header of the database is 00000000 53 51 4c 69 74 65 20 66 6f 72 6d 61 74 20 33 00 |SQLite format 3. and using sqlite3-cli it allows me to see actual message contents - just by opening the database. If SQLCipher was in use I wouldn't see that header/be able to extract the database I assume.

[justinclift]

@norpol Yeah, that sounds like the database you're trying there isn't encrypted. At least, not with SQLCipher.

[backuitist]

Thanks @justinclift good to know! I thought there would be some sort of encryption but couldn't find anything in the codebase, and the signal error message was exactly matching that of SQLite.

[GDmac]

On mac, as a work-around (loose everything / start from scratch) i deleted ApplicationSupport/Signal to be able to start the app at all.

[thrien]

I also have this issue (Arch) and downgrading the gtk3 to ..-3 "solved" the issue. Signal is starting again.

I did the same thing a few weeks ago but since this just deactivated encryption i now tried to find a better solution.
I know have the latest version of gtk-3 again and reencrypted the database in .config/Signal/sql/db.sqlite following this example using the key in .config/Signal/config.json.
Now my error message looks like this:

Unhandled Promise Rejection: Error: SqliteError: file is not a database
    at Database.pragma ([REDACTED]/app.asar.unpacked/ts/sql/mainWorker.bundle.js:714:27)
    at keyDatabase ([REDACTED]/app.asar.unpacked/ts/sql/mainWorker.bundle.js:27129:8)
    at openAndMigrateDatabase ([REDACTED]/app.asar.unpacked/ts/sql/mainWorker.bundle.js:27181:5)
    at openAndSetUpSQLCipher ([REDACTED]/app.asar.unpacked/ts/sql/mainWorker.bundle.js:27199:16)
    at Object.initialize ([REDACTED]/app.asar.unpacked/ts/sql/mainWorker.bundle.js:28423:14)
    at MessagePort.<anonymous> ([REDACTED]/app.asar.unpacked/ts/sql/mainWorker.bundle.js:30530:36)
    at MessagePort.[nodejs.internal.kHybridDispatch] (internal/event_target.js:354:41)
    at MessagePort.exports.emitMessage (internal/per_context/messageport.js:18:26)
    at Worker.<anonymous> ([REDACTED]/app.asar/ts/sql/main.js:33:29)
    at Worker.emit (events.js:315:20)
    at MessagePort.<anonymous> (internal/worker.js:207:53)
    at MessagePort.[nodejs.internal.kHybridDispatch] (internal/event_target.js:354:41)
    at MessagePort.exports.emitMessage (internal/per_context/messageport.js:18:26)

I get this wether or not i put the encrypted or not-encrypted file at .config/Signal/sql/db.sqlite. Since there are more databases i was wondering if they might need to be encrypted as well but when i have nothing at .config/Signal/sql/db.sqlite Signal starts without an error (as other people have already reported).
I don't understand why the error message changed but maybe someone else does and it might be helpful.

[primeos]

@thrien the linked Stack Overflow example uses a passphrase. For raw keys x'$KEY' is required (https://www.zetetic.net/sqlcipher/sqlcipher-api/#key). Maybe that's the issue?

You also need to manually copy the user_version (and schema_version) or Signal-Desktop will corrupt your DB on startup while performing all of the DB migrations.

In case someone's interested: I've wrote the following very hacky script that worked for me (but one needs to replace all of the @...@-placeholders with paths to the corresponding binaries): https://github.com/NixOS/nixpkgs/blob/45bd7b39a444c904986324b5f7c46ba867612575/pkgs/applications/networking/instant-messengers/signal-desktop/db-reencryption-wrapper.py

I don't understand why the error message changed but maybe someone else does and it might be helpful.

Not sure why it has changed but IIRC it changed in Signal-Desktop 5.1.0 (likely affects other error messages as well - probably an intentional change to provide additional context (stack trace) instead of just printing the "file is not a database" error message).

[LukeLR]

I already replied to the previous comment via e-Mail, but apparently GitHub has swallowed my reply once again :/ Here it is:

The issue was that some newer versions broke sqlcipher support, by downgrading, the sqlcipher support should have been restored. So if you downgraded to the correct, you should have been using an encrypted database. And by upgrading to a recent version, it should continue to work with that encrypted database.

However, did you try to encrypt the database using the CLI instead of using the API? I think there's less room for errors if you execute the SQL statements used for encryption directly on the CLI instead of wrapping them in the API. See also https://discuss.zetetic.net/t/how-to-encrypt-a-plaintext-sqlite-database-to-use-sqlcipher-and-avoid-file-is-encrypted-or-is-not-a-database-errors/868.

[kpcyrd]

I'm going to unsubscribe since the original issue has been resolved, if there's a new issue with the Arch Linux package please create an issue in the Arch Linux issue tracker.

This github issue can likely be closed.

[joukewitteveen]

I'm going to unsubscribe since the original issue has been resolved

@kpcyrd: I'm curious what the solution was. I can't find any change that indicates solving the static/dynamic linking issue, could you point to it?

[ghost]

I also haven't experienced this issue in a while on Arch Linux. Don't know if this is just a temporary condition or if the issue has been fixed. Thanks anyway for everyone involved with getting this resolved!

[cherti]

I also haven't experienced this issue in a while on Arch Linux. Don't know if this is just a temporary condition or if the issue has been fixed. Thanks anyway for everyone involved with getting this resolved!

It's a temporary fix. The gtk3-package wants to enable a feature regarding sqlite that clashes with the sqlcipher-implementation signal-desktop uses. Due to this, this feature in gtk3 was rolled back temporarily, but is still on the roadmap to be enabled again, whenever this will be. Eventually I guess, given the problem with sqlcipher (which is: sqlcipher disguises itself as sqlite, to be a drop-in replacement, but the gtk3-feature will load sqlite early on (i.e. before the application itself comes up) and then sqlcipher's intended place is already occupied by normal sqlite, which then cannot read the database, as it is an sqlcipher-one, hence the error.)

[thrien]

@thrien the linked Stack Overflow example uses a passphrase. For raw keys x'$KEY' is required (https://www.zetetic.net/sqlcipher/sqlcipher-api/#key). Maybe that's the issue?

You also need to manually copy the user_version (and schema_version) or Signal-Desktop will corrupt your DB on startup while performing all of the DB migrations.

Holy shit it works again. Thank you very much!
I used sqlcipher on the command line to encrypt the database.

$ sqlcipher db.sqlite
sqlite> PRAGMA user_version;
23
sqlite> PRAGMA schema_version;
113
sqlite> ATTACH DATABASE 'db.encrypted' AS encrypted KEY "x'<key from config.json>'";
sqlite> SELECT sqlcipher_export('encrypted');
sqlite> DETACH DATABASE encrypted;
sqlite> .quit;
$ sqlcipher db.encrypted
sqlite> PRAGMA key = "x'<key from config.json>'";
sqlite> PRAGMA user_version = 23;
sqlite> PRAGMA schema_version = 113;
sqlite> .quit

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[norpol]

I mean people lost their signal history, there hasn't been any comment whether this has actually been fixed + the bug of deleting the database when you click the wrong button is likely also not fixed?

[kpcyrd]

The title describes a symptom and different people joined this issue for different root causes.

Arch Linux currently patches one of the Signal-Desktop dependencies for gnome support, the pull request to upstream the change is at signalapp/better-sqlite3#3.

The "database deletion when closing the dialog" bug should be tracked in a different github issue in case it's still present.

[Tmpod]

Ran into the same issue. Decided I didn't want to delete my data and investigate first whether there was a solution.

Clicked the "X" in the window decoration of the error popup - all my data got purged. Thanks a lot Signal team for your excellent error handling and backup strategy.

Exact same thing happened to me. ~3 years worth of chats gone, and unfortunately my last backup is pretty much 1y old... Ran photorec on my whole disk and recovered a lot of files, but now sieving through them will take ages, with no guarantee I can even recover the database properly.
Now my only source for my messages is my phone, but seeing as I can't export to an external SD card or bind two directories from the two filesystems, I simply don't have space to perform a 10GB export...

Honestly getting really fed up about this whole anti-backup anti-phone-sync attitude from Signal.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.