[Bug]: database is locked

[auipga]

What happened?

Regularly reading this error while changing nothing and just rerunning the deployment:

error returned from database: (code: 5) database is locked

Version

v0.24.0

Which operating system(s) are you seeing the problem on?

In deployment

Which CPU architectures are you seeing the problem on?

In deployment

Relevant log output

cargo shuttle logs *****

 INFO Entering queued state
TRACE shuttle_deployer::deployment::queue: getting a build slot
TRACE {uri="http://gateway:8001/stats/load"} shuttle_deployer::deployment::gateway_client: calling gateway
DEBUG hyper::client::connect::http: connecting to 10.99.0.14:8001
DEBUG hyper::client::connect::http: connected to 10.99.0.14:8001
DEBUG hyper::client::pool: pooling idle connection for ("http", gateway:8001)
TRACE {response="Response { status: 200, version: HTTP/1.1, headers: {\"content-type\": \"application/json\", \"content-length\": \"38\", \"date\": \"Tue, 05 Sep 2023 21:59:58 GMT\"}, body: Body(Streaming) }"} shuttle_deployer::deployment::gateway_client: Load response

 INFO Entering building state
 INFO     Finished release [optimized] target(s) in 0.17s

 INFO Entering crashed state
ERROR {error="Build error: Database error: error returned from database: (code: 5) database is locked"} shuttle_deployer::deployment::queue: service build encountered an error

Duplicate declaration

• I have searched the issues and there are none like this.

[auipga]

Same with v0.25.0

[oddgrd]

Hey @auipga, we have a ticket for this here: #1067, but perhaps it's not so easy to find without "database is locked" in the title. Did you run just a single deployment, or did you attempt several? We're working on resolving the issue, but in the meantime you should be able to avoid the bug by letting a deployment finish before starting a new one (which will be the case if you don't exit the deployment stream). If you do end up with multiple deployments in the building/loading states, you should be able to reset it with a cargo shuttle project restart. Sorry for the trouble!

[auipga]

cargo deploy I ran a single deployment until it crashed.
cargo logs ... I read the logs.
update the code
cargo deploy rerun the deployment.

Although waiting a minute or more between the deployments reduced the likelihood of the mentioned error it still occurred sometimes. (It drove me a little crazy because I changed things back and forth but getting inconsistent results)

I read #1067 back when I created this issue but I thought that a crashed deployment is not running anymore hence no concurrency.

I'll try your suggestion to cargo shuttle project restart and let you know if it helps.

[oddgrd]

Ah, okay, thanks for the info! It does seem like this error can occur when having an active deployment and one building too, which is unexpected. We're about to make a change in the coming week that will stop writing logs to the deployer sqlite database, this should alleviate this issue since logs are the majority of writes to the DB. In the slightly longer term we aim to completely remove this DB from deployers (the container your service runs in), and of course support interuptible builds as stated in #1067.

[jonaro00]

Although this bug is more rare now, it can still happen.

[elbaro]

Installed cargo-shuttle for the first time.
cargo shuttle init --template axum hello-world followed by cargo shuttle login, cargo shuttle deploy crashed with this error.

Also project restart should accept --allow-dirty.

❯ cargo shuttle project restart
Project "helo" is destroyed
Run `cargo shuttle project start` to recreate project environment on Shuttle.
Project "helo" is ready
Your project will sleep if it is idle for 30 minutes.
To change the idle time refer to the docs: https://docs.shuttle.rs/getting-started/idle-projects

Run `cargo shuttle deploy --allow-dirty` to deploy your Shuttle service.

❯ cargo shuttle project restart --allow-dirty
error: unexpected argument '--allow-dirty' found

Usage: cargo-shuttle project restart [OPTIONS]

[auipga]

@elbaro

1. hello-world != helo
2. Output: Run cargo shuttle project start
   You ran: cargo shuttle project **re**start + --allow-dirty, why?
3. --allow-dirty is for deployment.

   $ cargo shuttle deploy -h
   ...
   Options:
         --allow-dirty
             Allow deployment with uncommitted files [aliases: ad]
   ...

4. cargo shuttle project restart has no --allow-dirty option. It would make no sense to have this option for restart. Maybe I did't get the point, so please explain in more detail.
5. Your issue seems not to be linked to this specific issue, so please open a new issue for your situation.

[elbaro]

Well the first project start gave [Bug]: database is locked and the second project start gave it's already running error. I ran restart to clear any corrupted state. The dashboard page doesn't indicate it's running and I couldn't access the deployed page even after a few restarts. I am providing a data point where the cloud state might be corrupted from this bug.

--allow-dirty is a different issue as you said and probably I mixed the concept of deploy and start at the moment.

[jonaro00]

This bug has become a lot more common now and we have raised the priority to solve it.

[lazyfuhrer]

2024-01-03T03:55:09.866+05:30 [Deployer]  INFO Starting service
2024-01-03T03:55:14.870+05:30 [Deployer]  INFO Entering Crashed state
2024-01-03T03:55:14.870+05:30 [Deployer]  INFO Cleaning up crashed deployment
2024-01-03T03:55:14.870+05:30 [Deployer] ERROR shuttle_deployer::deployment::run: {error="Run error: errored while setting the new deployer address: Database error: error returned from database: (code: 5) database is locked"} Service encountered an error and crashed

Any idea how can I fix this ?

[jonaro00]

The solution is usually to restart the project and deploy again.

[jonaro00]

Update: This bug has been hard to reproduce, but it looks to be less frequent now than in previous weeks. We are also working on a bigger change that should eliminate this bug once and for all (getting rid of the sqlite dbs in deployers).

[gnomeria]

I just got realized I got this issue. Even after a cargo shuttle project restart it kept erroring out. :(
Now after the second project restart, the status has not changed from This project is booting up, please refresh the page in a few seconds for more than 15 minutes. Did the 3rd restart, still back to original error:

2024-02-03T00:11:50.929698Z [Deployer]  INFO shuttle_deployer::deployment::run: successfully loaded service
2024-02-03T00:11:50.929734Z [Deployer]  INFO shuttle_deployer::persistence: Uploading resources to resource-recorder
2024-02-03T00:11:50.930351Z [Deployer]  INFO Entering Running state
2024-02-03T00:11:50.930353Z [Deployer]  INFO Starting service
2024-02-03T00:11:55.934314Z [Deployer]  INFO Entering Crashed state
2024-02-03T00:11:55.934325Z [Deployer]  INFO Cleaning up crashed deployment
2024-02-03T00:11:55.934364Z [Deployer] ERROR shuttle_deployer::deployment::run: {error="Run error: errored while setting the new deployer address: Database error: error returned from database: (code: 5) database is locked"} Service encountered an error and crashed

I tried to update the cargo-runtime and cargo-axum to 0.38.0 and did another redeploy, it worked.
But then after updating the code and redeployed, got the original issue again, with several attempts at cargo shuttle project restart to no avail 😞

[jonaro00]

We have continued investigating this, and we made some improvements in 0.39.0 that we hope help reduce the frequency of this bug.

[jonaro00]

The probability of getting this error seems to be reduced since 0.39.0

[jonaro00]

Improvements in recent releases has made this error very rare now. We are continuing work on getting rid of it completely.