Getting non-matching LTX checksum on fresh volume

[kentcdodds]

https://github.com/kentcdodds/kentcdodds.com/actions/runs/3316512422/jobs/5478478215

cannot open store: open databases: open database("sqlite.db"): verify database file: database checksum (e3d3906d74cc0273) does not match latest LTX checksum (0000000000000000)

This volume is brand new and completely empty. @benbjohnson said this is a bug that needs fixing and asked me to open this issue. More context at https://www.youtube.com/watch?v=vTNPJGKqsYQ

Thanks!

[benbjohnson]

@kentcdodds Thanks for writing this up. I realized I had an old version (pr-109) still on the litefs-example which it looks like you have in your Dockerfile as well. Sorry about that. Can you try changing this line here to:

FROM flyio/litefs:0.2 AS litefs

I'm surprised to see that error on a brand new volume as it happens when LiteFS is validating the existing database state. Can you retry with the new litefs version and let me know if you still have the same issue?

[kentcdodds]

Thanks! I've still go the same issue: https://github.com/kentcdodds/kentcdodds.com/actions/runs/3322212924/jobs/5491067040

[kentcdodds]

I'm a bit stuck on deploying LiteFS until this is resolved. Any ideas?

[benbjohnson]

@kentcdodds The error is strange because it is essentially saying that the database state on disk exists (checksum e3d3906d74cc0273) but there's no associated replication data (checksum 0000000000000000). However, you're seeing that error even when you deploy with a clean volume so there shouldn't be any database state.

2022-10-25T15:35:44Z   [info]cannot open store: open databases: open database("sqlite.db"): verify database file: database checksum (e3d3906d74cc0273) does not match latest LTX checksum (0000000000000000)

Can you try removing the volumes on your staging set up and re-deploying and seeing if you still have the same error?

[kentcdodds]

I think I've figured out what's going on. When I create the new volume, my old (pre litefs) app restarts and applies migrations to the new db in the volume which is what causes this issue.

What I'm trying now is to deploy a version of my app that does not do anything to the database so then I can have that one running when I recreate the volume, and then deploy the litefs version. Will let you know what happens.

[benbjohnson]

Ok, cool. Thanks for digging into it more. I also created an issue for keeping litefs running on error so it's easier to ssh in and debug the state. #136

[benbjohnson]

I pushed up a PR for it so it's available at pr-137 in Docker now. That'll keep litefs running even if it hits some kind of error on startup so the fly instance will be accessible via ssh.

[kentcdodds]

Good news! It's running now.

Now I'm going to try to create more regions. It just occurred to me that I'll need to create volumes for the regions first right? If I try to deploy my app to a region without a persistent volume things will break right?

[kentcdodds]

Interestingly, I added a volume to maa, and then tried to add a region there and got this error message:

Error App 'kcd-staging' uses volumes to control regions. Add or remove volumes to change region placement.

So I just scaled up to a count of 2 and maa started right up! Wahoo! Thanks a ton for the help!

Now I just need to figure out how to determine the primary region via that .primary file and then I think I should be ready to go with this to prod!

[kentcdodds]

I got this again on a new deploy of the app:

2022-10-25T22:33:59.223 app[18d0f7c2] den [info] ERROR: cannot open store: open databases: open database("sqlite.db"): verify database file: database checksum (f13013272ddb586c) does not match latest LTX checksum (da9624ecbb43ad42)

I'm not sure what I'm doing wrong :(

[kentcdodds]

Here's the failed build, not sure how useful it'll be: https://github.com/kentcdodds/kentcdodds.com/actions/runs/3324709637/jobs/5496681848

[benbjohnson]

@kentcdodds This is a known bug that can occur on restart with the rollback journal. I have a fix for this one. We should have a v0.3.0 release coming early next week that will have WAL support and stability fixes in it.