New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store Build Number in repository table #1919

Closed
naveensrinivasan opened this Issue Jan 29, 2017 · 4 comments

Comments

Projects
3 participants
@naveensrinivasan

naveensrinivasan commented Jan 29, 2017

There are webhook 500 errors with

"time=\"2017-01-11T15:28:54Z\" level=error msg=\"failure to save commit for Revature/core. meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '425-2' for key 'build_number'\" \n"
"time=\"2017-01-11T15:28:54Z\" level=error msg=\"Error #01: meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '425-2' for key 'build_number'\\n\" ip=10.138.0.10 latency=1.472476484s method=POST path=\"/hook\" status=500 time=\"2017-01-11T15:28:54Z\" user-agent=\"GitHub-Hookshot/31cb34a\" \n"
"time=\"2017-01-11T15:40:36Z\" level=error msg=\"failure to save commit for Revature/core. meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '429-2' for key 'build_number'\" \n"
"time=\"2017-01-11T15:40:36Z\" level=error msg=\"Error #01: meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '429-2' for key 'build_number'\\n\" ip=10.8.0.1 latency=1.73831831s method=POST path=\"/hook\" status=500 time=\"2017-01-11T15:40:36Z\" user-agent=\"GitHub-Hookshot/31cb34a\" \n"
"time=\"2017-01-11T15:41:48Z\" level=error msg=\"failure to save commit for Revature/core. meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '430-2' for key 'build_number'\" \n"
"time=\"2017-01-11T15:41:48Z\" level=error msg=\"Error #01: meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '430-2' for key 'build_number'\\n\" ip=10.138.0.10 latency=1.493025308s method=POST path=\"/hook\" status=500 time=\"2017-01-11T15:41:48Z\" user-agent=\"GitHub-Hookshot/31cb34a\" \n"
"time=\"2017-01-11T15:54:37Z\" level=error msg=\"Error #01: sql: no rows in result set\\n\" ip=10.8.0.1 latency=463.042715ms method=GET path=\"/api/repos/Revature/frontend/logs/209/1\" status=404 time=\"2017-01-11T15:54:37Z\" user-agent=\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36\" \n"
"time=\"2017-01-11T15:54:47Z\" level=error msg=\"Error #01: sql: no rows in result set\\n\" ip=10.8.0.1 latency=489.376739ms method=GET path=\"/api/repos/Revature/frontend/logs/208/1\" status=404 time=\"2017-01-11T15:54:47Z\" user-agent=\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36\" \n"
"time=\"2017-01-11T17:34:42Z\" level=error msg=\"failure to save commit for Revature/core. meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '435-2' for key 'build_number'\" \n"
"time=\"2017-01-11T17:34:42Z\" level=error msg=\"Error #01: meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '435-2' for key 'build_number'\\n\" ip=10.138.0.10 latency=1.310611147s method=POST path=\"/hook\" status=500 time=\"2017-01-11T17:34:42Z\" user-agent=\"GitHub-Hookshot/31cb34a\" \n"
"time=\"2017-01-11T19:12:27Z\" level=error msg=\"failure to save commit for Revature/frontend. meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '212-3' for key 'build_number'\" \n"
"time=\"2017-01-11T19:12:27Z\" level=error msg=\"Error #01: meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '212-3' for key 'build_number'\\n\" ip=10.138.0.10 latency=1.387678893s method=POST path=\"/hook\" status=500 time=\"2017-01-11T19:12:27Z\" user-agent=\"GitHub-Hookshot/31cb34a\" \n"
"time=\"2017-01-11T20:19:28Z\" level=error msg=\"failure to save commit for Revature/frontend. meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '213-3' for key 'build_number'\" \n"
"time=\"2017-01-11T20:19:28Z\" level=error msg=\"Error #01: meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '213-3' for key 'build_number'\\n\" ip=10.8.0.1 latency=1.39787868s method=POST path=\"/hook\" status=500 time=\"2017-01-11T20:19:28Z\" user-agent=\"GitHub-Hookshot/31cb34a\" \n"
"time=\"2017-01-11T20:20:52Z\" level=error msg=\"failure to save commit for Revature/frontend. meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '216-3' for key 'build_number'\" \n"
"time=\"2017-01-11T20:20:52Z\" level=error msg=\"Error #01: meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '216-3' for key 'build_number'\\n\" ip=10.138.0.10 latency=1.41621908s method=POST path=\"/hook\" status=500 time=\"2017-01-11T20:20:52Z\" user-agent=\"GitHub-Hookshot/31cb34a\" \n"
"time=\"2017-01-11T21:42:07Z\" level=error msg=\"failure to save commit for Revature/core. meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '436-2' for key 'build_number'\" \n"
"time=\"2017-01-11T21:42:07Z\" level=error msg=\"Error #01: meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '436-2' for key 'build_number'\\n\" ip=10.138.0.10 latency=1.481351984s method=POST path=\"/hook\" status=500 time=\"2017-01-11T21:42:07Z\" user-agent=\"GitHub-Hookshot/6f33334\" \n" 

The most of the http 500 are because of duplicate Error #01: meddler.Insert: DB error in Exec: Error 1062: Duplicate entry. How is this possible and the funny thing is that If I manualy redeliver the webhook it does not fail and returns with http 200.

For example here is a github webhook which is failing with id 6f33334

and here is the corresponding error for that in the logs

"time=\"2017-01-11T21:42:07Z\" level=error msg=\"Error #01: meddler.Insert: DB error in Exec: Error 1062: Duplicate entry '436-2' for key 'build_number'\\n\" ip=10.138.0.10 latency=1.481351984s method=POST path=\"/hook\" status=500 time=\"2017-01-11T21:42:07Z\" user-agent=\"GitHub-Hookshot/6f33334\" \n"

This issue has been discussed here http://discourse.drone.io/t/webhook-500-errors/151/7

@andreas-venturini

This comment has been minimized.

Show comment
Hide comment
@andreas-venturini

andreas-venturini Mar 8, 2017

Also observing this infrequently

time="2017-03-08T11:05:00Z" level=error msg="failure to save commit for GITHUB_ORG/GITHUB_PROJECT. meddler.Insert: DB error in Exec: UNIQUE constraint failed: builds.build_number, builds.build_repo_id" 
time="2017-03-08T11:05:00Z" level=error msg="Error #01: meddler.Insert: DB error in Exec: UNIQUE constraint failed: builds.build_number, builds.build_repo_id\n" ip=192.30.252.42 latency=2.142335821s method=POST path="/hook" status=500 time="2017-03-08T11:05:00Z" user-agent="GitHub-Hookshot/6b02022" 

Redelivering the payload via GitHub GUI always works.
Didn't have time to investigate. Might be related to the size of the SQLite database file (meaning the larger the database file the more likely for the error to occur). Deleting the database file and re-creating the GitHub webhooks seems to solve the issue for me.

andreas-venturini commented Mar 8, 2017

Also observing this infrequently

time="2017-03-08T11:05:00Z" level=error msg="failure to save commit for GITHUB_ORG/GITHUB_PROJECT. meddler.Insert: DB error in Exec: UNIQUE constraint failed: builds.build_number, builds.build_repo_id" 
time="2017-03-08T11:05:00Z" level=error msg="Error #01: meddler.Insert: DB error in Exec: UNIQUE constraint failed: builds.build_number, builds.build_repo_id\n" ip=192.30.252.42 latency=2.142335821s method=POST path="/hook" status=500 time="2017-03-08T11:05:00Z" user-agent="GitHub-Hookshot/6b02022" 

Redelivering the payload via GitHub GUI always works.
Didn't have time to investigate. Might be related to the size of the SQLite database file (meaning the larger the database file the more likely for the error to occur). Deleting the database file and re-creating the GitHub webhooks seems to solve the issue for me.

@bradrydzewski

This comment has been minimized.

Show comment
Hide comment
@bradrydzewski

bradrydzewski May 1, 2017

Member

The root cause is when two webhooks are received at the same time and drone tries to increment and use the same build number for each, thus violating the database unique constraint. No need to delete the database when this happens, just resent the webhook.

A fix is planned, but not for this release cycle.

I will update the issue when there is progress to share.

Member

bradrydzewski commented May 1, 2017

The root cause is when two webhooks are received at the same time and drone tries to increment and use the same build number for each, thus violating the database unique constraint. No need to delete the database when this happens, just resent the webhook.

A fix is planned, but not for this release cycle.

I will update the issue when there is progress to share.

@drone drone locked and limited conversation to collaborators May 1, 2017

@bradrydzewski bradrydzewski added this to To Do in Version 0.7 May 11, 2017

@bradrydzewski bradrydzewski changed the title from Webhook 500 errors - DB error in Exec: Error 1062: Duplicate entry to Store Build Number in repository table May 11, 2017

@bradrydzewski

This comment has been minimized.

Show comment
Hide comment
@bradrydzewski

bradrydzewski May 11, 2017

Member

I'm renaming this issue to reflect the proposed solution for 0.7. Instead of doing a select max(build_number) to get the next sequential build number, we will store the build number in the repository table.

To increment we will execute the following sql statement:

update repos set repo_build_seq=? where repo_build_seq=?

If the build number fails to increment and has already been incremented by another go routine, it will returns a result set indicating zero rows were updated, and we can retry. Once the build number is acquired it can be used to persist the build.

Member

bradrydzewski commented May 11, 2017

I'm renaming this issue to reflect the proposed solution for 0.7. Instead of doing a select max(build_number) to get the next sequential build number, we will store the build number in the repository table.

To increment we will execute the following sql statement:

update repos set repo_build_seq=? where repo_build_seq=?

If the build number fails to increment and has already been incremented by another go routine, it will returns a result set indicating zero rows were updated, and we can retry. Once the build number is acquired it can be used to persist the build.

@bradrydzewski bradrydzewski moved this from To Do to In Progress in Version 0.7 May 16, 2017

@bradrydzewski bradrydzewski moved this from In Progress to Done in Version 0.7 May 23, 2017

@bradrydzewski

This comment has been minimized.

Show comment
Hide comment
@bradrydzewski

bradrydzewski May 23, 2017

Member

closed by #2055

Member

bradrydzewski commented May 23, 2017

closed by #2055

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.