New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

random database lock when sqlite database #2040

Open
lunny opened this Issue Jun 23, 2017 · 24 comments

Comments

10 participants
@lunny
Copy link
Member

lunny commented Jun 23, 2017

This is a summary issue since there are so many issues to report the database lock when using sqlite. I will close all other issues and please continue in this issue.

@lunny

This comment has been minimized.

Copy link
Member

lunny commented Jun 23, 2017

And maybe we should pause make test-sqlite util this issue is resolved.

@typeless

This comment has been minimized.

Copy link
Contributor

typeless commented Jun 27, 2017

@bkcsoft

This comment has been minimized.

Copy link
Member

bkcsoft commented Jun 28, 2017

@typeless Unless that is available in https://github.com/mattn/go-sqlite3 and https://github.com/go-xorm/xorm/ we can't use it 😞

@typeless

This comment has been minimized.

Copy link
Contributor

typeless commented Jun 29, 2017

@bkcsoft Agreed. This would probably need to be fixed upstream.

@andreynering

This comment has been minimized.

Copy link
Member

andreynering commented Jul 5, 2017

I hope #2116 fixes the issue

@lunny lunny closed this in #2116 Jul 6, 2017

lunny added a commit that referenced this issue Jul 6, 2017

@andreynering

This comment has been minimized.

Copy link
Member

andreynering commented Jul 6, 2017

Reopening, since it seems it not really fixed.

@andreynering andreynering reopened this Jul 6, 2017

lafriks added a commit to lafriks/gitea that referenced this issue Jul 12, 2017

@lunny lunny modified the milestones: 1.3.0, 1.4.0 Oct 25, 2017

@lafriks lafriks modified the milestones: 1.4.0, 1.5.0 Jan 14, 2018

@lunny lunny modified the milestones: 1.5.0, 1.6.0 May 11, 2018

@PaulBINK

This comment has been minimized.

Copy link

PaulBINK commented Jun 18, 2018

We are getting this problem frequently when working with issues: add label, comment, close issue, reopen. Sooner or later it will 500 on us. According to the logs it happens when calling GetNotificationCount.

@jonasfranz

This comment has been minimized.

Copy link
Member

jonasfranz commented Jun 18, 2018

Enabling WAL mode worked for me. https://www.sqlite.org/wal.html

@hoxnox

This comment has been minimized.

Copy link

hoxnox commented Aug 30, 2018

Same for me.

Caused by: hudson.plugins.git.GitException: Command "/usr/local/git/bin/git fetch --tags --progress ssh://sources.acme.com/repo.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: Gitea: Internal error
Failed to get repository: database is locked
@typeless

This comment has been minimized.

Copy link
Contributor

typeless commented Aug 31, 2018

@bvinc

This comment has been minimized.

Copy link

bvinc commented Sep 1, 2018

Author of go-sqlite-lite here. My driver, and also crawshaw's driver, don't provide a database/sql interface, so they would be extremely inconvenient to use. But for a few reasons, it would probably help with locking errors. My driver gives complete control over connections, which means you'll only get locking errors if you're doing something directly to cause them. The crawshaw driver defaults to using shared cache mode and WAL mode, and uses the unlock_notify API to prevent locking issues with shared cache mode.

I think the current mattn driver is the best one to use if you require a database/sql driver. Their FAQ mentions setting the the maximum number of open connections to 1. Have you tried that?

https://github.com/mattn/go-sqlite3#faq

@lunny

This comment has been minimized.

Copy link
Member

lunny commented Sep 2, 2018

@bvinc thanks for your reply, It's a good idea to send a PR to set open connections to 1 when using mattn/go-sqlite3

@hoxnox

This comment has been minimized.

Copy link

hoxnox commented Sep 4, 2018

What is the best way to migrate from sqlite to postgresql?

@typeless

This comment has been minimized.

Copy link
Contributor

typeless commented Oct 21, 2018

Now that mattn/go-sqlite3#439 is merged, we can either use the tip right away or wait for the new release of go-sqlite3 to fix the issue. Note that to enable this functionality, we have to add a new build tag sqlite_unlock_notify.

Edit: I hope the users don't have to specify -tags sqlite sqlite_unlock_notify besides just -tags sqlite in that it would be better to not expose the nuances of a dependent package. I don't know how it should be done though.

Edit2: I changed my mind. If one wants to build Gitea from source, he should be capable of handling the additional flag. We should update the documentation accordingly though.

@lunny

This comment has been minimized.

Copy link
Member

lunny commented Oct 21, 2018

at first, we need a PR to update go-sqlite3 library.

@lunny

This comment has been minimized.

Copy link
Member

lunny commented Oct 23, 2018

Even #5144 resolved single process issue. There is some issues when pushing via SSH. This should be closed by #4886.

@lunny lunny reopened this Oct 23, 2018

@sapk

This comment has been minimized.

Copy link
Member

sapk commented Oct 23, 2018

@typeless go-sqlite3 library with sqlite_unlock_notify tag seems to not build on win 32bit https://drone.gitea.io/go-gitea/gitea/3536/18

@typeless

This comment has been minimized.

Copy link
Contributor

typeless commented Oct 23, 2018

@sapk Similar to mattn/go-sqlite3#238. Will submit a fix soon.

@sapk

This comment has been minimized.

Copy link
Member

sapk commented Oct 23, 2018

@typeless exactly what I was searching but I use the terms uint too large and I found nothing ...

@typeless

This comment has been minimized.

Copy link
Contributor

typeless commented Oct 23, 2018

I have submitted mattn/go-sqlite3#654.
However, I cannot find a computer with Windows/386 to test. 😨

@sapk

This comment has been minimized.

Copy link
Member

sapk commented Oct 23, 2018

@typeless use gox to cross-compile ?

@typeless

This comment has been minimized.

Copy link
Contributor

typeless commented Oct 24, 2018

@sapk I just installed it. But after typing gox -osarch="windows/linux" -tags sqlite_unlock_notify
, it only prints Number of parallel builds: 31. I've not figured out how to use it to run go test either.

Edit: fixed at mattn/go-sqlite3#655

@lunny

This comment has been minimized.

Copy link
Member

lunny commented Oct 24, 2018

Just type make release-windows on gitea?

@typeless

This comment has been minimized.

Copy link
Contributor

typeless commented Oct 24, 2018

#5162 fixes the CI failure and TAGS='bindata sqlite sqlite_unlock_notify' make release-windows passes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment