Retry upserts that fail with "duplicate key error" #291

babbageclunk · 2016-06-27T17:15:06Z

According to the Mongo docs (link below), getting this error from an upsert is expected
behaviour and the client should retry if it happens.
https://docs.mongodb.com/v3.2/reference/method/db.collection.update/#use-unique-indexes

It doesn't seem to be specific to Mongo 3.2, but I think that the performance characteristics have changed in a way that makes it much more common than it is with 2.x.

Looking through the code revealed that removes had the same problem - I verified this by modifying the repro code to do removes instead.

This fixes https://github.com/go-mgo/mgo/issues/277

According to the Mongo docs (link below), getting this error is expected behaviour and the client should retry if it happens. https://docs.mongodb.com/v3.2/reference/method/db.collection.update/#use-unique-indexes This fixes https://github.com/go-mgo/mgo/issues/277

niemeyer · 2016-06-28T06:14:42Z

Any chance we can get a test?

Also, wonder if we should be retrying on mgo itself. Any reason not to?

babbageclunk · 2016-06-28T12:39:13Z

Good point, the retry should definitely be in mgo, rather than mgo.txn. I'll move it. Are you alright with the infinite loop to do the retry? Or should I be doing something a bit cleverer?

Sorry, I should have added a test - it's fiddly to do, but it looks like there are already some other tests trying to reproduce racy things.

Also, use IsDup to detect the duplicate key errors - it looks like that's been accreted through painful experience.

babbageclunk · 2016-07-05T15:25:00Z

Hi, sorry for the delay on this - I've moved the retrying into mgo, in Query.Apply and Collection.Upsert. Unfortunately I've had real trouble reproducing the problem in a test at that level - no matter how I try I can't get the upserts to fail with an 11000 error.

However when I run the txn tests TestTxnQueueStashStressTest was reliably failing and now passes consistently with the change. (I don't really understand why the txn tests aren't run by go test ./... from the mgo directory.)

babbageclunk · 2016-07-05T21:32:23Z

It seems like the -check.v in .travis.yml means the mgo/txn tests aren't being run - is that deliberate? I've added a line to run the txn tests as well.

babbageclunk · 2016-07-06T07:51:11Z

Hmm - that's unfortunate. It looks like TestTxnQueueStressTest fails intermittently on the parent branch as well. I'm going to pull the change to turn on the mgo/txn tests out of this PR into a new one and add a skip for that test until someone can work out what's going wrong.

babbageclunk · 2016-07-06T09:26:21Z

Sigh - no, of course, that doesn't work because this PR fixes TestTxnQueueStashStressTest. So I'm including the skip here.

At the moment it fails about 25% of the time - the reason needs to be worked out, but it's probably better to get the txn tests running first.

It fails sometimes in a way that seems like a timing issue - put a sleep in before retrying (up to 3 attempts).

If 10 1-second waits aren't enough then it's likely that the "no reachable servers" error isn't transient.

Make StartServer dial the server first to ensure it's up before it returns. Error logging showed that something else was already using port 50017. Picking a random port still gets collisions :(, so try picking an unused one instead.

babbageclunk · 2016-07-06T19:48:59Z

To show that this change does fix the duplicate key error, here's a build of a branch that has the txn tests turned on without the upsert retrying - TestTxnQueueStashStressTest fails in the mongo 3.2 test run.

niemeyer · 2016-07-08T18:09:19Z

session.go

@@ -2484,7 +2484,15 @@ func (c *Collection) Upsert(selector interface{}, update interface{}) (info *Cha
 		Flags:      1,
 		Upsert:     true,
 	}
-	lerr, err := c.writeOp(&op, true)
+	var lerr *LastError
+	for {


Let's please start by retrying at most 5 times here and in the other loop. More than that there's a good chance that there's something else wrong.

babbageclunk · 2016-07-10T21:08:38Z

Thanks for going through this!

This has better handling of used ports and error reporting. Remove mgo_test.go, it's not needed now. sim_test.go:simulate was relying on the mgoaddr global set in mgo_test.go, changed it to get the DBServer passed in.

This is needed so the txn tests can use dbtest.

babbageclunk · 2016-07-10T22:23:38Z

This is a spurious failure - the script to start the various mongo processes and set up users failed with couldn't add user: not master
The same commit passed on Travis here: https://travis-ci.org/babbageclunk/mgo/builds/143769207

rogpeppe · 2016-07-12T14:01:35Z

session.go

+
+		if err == nil {
+			break
+		} else if change.Upsert && IsDup(err) {


given that the else is redundant, perhaps leave it out and put the if on a new line?

Removing those two elses and making the ifs independent reads much better, thanks

rogpeppe · 2016-07-12T14:08:24Z

LGTM FWIW. What an unfortunate semantic.

rogpeppe · 2016-07-12T14:17:00Z

BTW it would be great if this could land soon, as it fixes a current critical bug in juju.

jameinel · 2016-07-14T09:06:57Z

session.go

@@ -2484,7 +2487,15 @@ func (c *Collection) Upsert(selector interface{}, update interface{}) (info *Cha
 		Flags:      1,
 		Upsert:     true,
 	}
-	lerr, err := c.writeOp(&op, true)
+	var lerr *LastError
+	for i := 0; i < maxUpsertRetries; i++ {


I believe mgo has an ability to hook in a log function, can we get retries like this into that mechanism so we have an idea how often it might be triggering.
Maybe something like logging only on success if i > 0 ?

Clarify the retry logic in Query.Apply.

Clarify what happens when test fails and what it might mean.

niemeyer · 2016-07-14T18:58:23Z

session.go

+		// https://docs.mongodb.com/v3.2/reference/method/db.collection.update/#use-unique-indexes
+		if !IsDup(err) {
+			if i > 0 {
+				debugf("upsert retry succeeded after %d failure(s)", i)


The error message is bogus. Not having a dup error does not mean the upsert succeeded.

Please drop the debug message altogether. If we have debug on, we'll see the attempts going through.

niemeyer · 2016-07-14T19:07:57Z

Merging, will fix that afterwards.

babbageclunk · 2016-07-14T19:54:52Z

Thanks!

babbageclunk · 2016-07-14T20:04:58Z

I've got a branch with these same changes against the v2 branch. Do you want me to make another PR for that, or will you just merge this across to there?

This version includes the fixes for the duplicate key error on upsert, which means we don't need the patch any more. go-mgo/mgo#291 go-mgo/mgo#316

Update mgo dependency This version includes the fixes for the duplicate key error on upsert, which means we don't need the patch any more. [go-mgo/mgo#291](go-mgo/mgo#291) [go-mgo/mgo#316](go-mgo/mgo#316) Add a readme to keep the patches directory alive and explain its use. mgz has made a change to apply_patches.py to ignore any files with a different extension. (Review request: http://reviews.vapour.ws/r/5477/)

babbageclunk added 2 commits June 27, 2016 17:51

Tweak formatting

cd7f33e

babbageclunk added 3 commits June 28, 2016 14:42

Move upsert-retrying into Query.Apply

ffc620c

Retry on duplicate key errors in Collection.Upsert

88b327f

Also, use IsDup to detect the duplicate key errors - it looks like that's been accreted through painful experience.

Remove unused code

78c6063

Run txn tests in Travis config

b59bff9

babbageclunk added 4 commits July 6, 2016 10:26

Skip TestTxnQueueStressTest unless -flaky is passed

2a98e8c

At the moment it fails about 25% of the time - the reason needs to be worked out, but it's probably better to get the txn tests running first.

Retry DropAll in txn test setup

6d88d84

It fails sometimes in a way that seems like a timing issue - put a sleep in before retrying (up to 3 attempts).

Bumping up retries and sleep time

ddea9f6

If 10 1-second waits aren't enough then it's likely that the "no reachable servers" error isn't transient.

Try to avoid port collisions when starting txn tests

e6ca633

Make StartServer dial the server first to ensure it's up before it returns. Error logging showed that something else was already using port 50017. Picking a random port still gets collisions :(, so try picking an unused one instead.

niemeyer reviewed Jul 8, 2016
View reviewed changes

Review changes from Gustavo

11b8488

babbageclunk added 2 commits July 10, 2016 22:58

Change txn tests to use dbtest instead of mgo_test

eba0955

This has better handling of used ports and error reporting. Remove mgo_test.go, it's not needed now. sim_test.go:simulate was relying on the mgoaddr global set in mgo_test.go, changed it to get the DBServer passed in.

Install gopkg.in/tomb.v2 in Travis CI

f253ba2

This is needed so the txn tests can use dbtest.

rogpeppe reviewed Jul 12, 2016
View reviewed changes

jameinel reviewed Jul 14, 2016
View reviewed changes

babbageclunk added 2 commits July 14, 2016 13:34

Review change from rogpeppe

3529517

Clarify the retry logic in Query.Apply.

Review change from rogpeppe

28af4c4

Clarify what happens when test fails and what it might mean.

babbageclunk force-pushed the fix-277 branch from bbc34d0 to 28af4c4 Compare July 14, 2016 12:37

Review change from jam - log if there was a retry

947661c

niemeyer reviewed Jul 14, 2016
View reviewed changes

niemeyer merged commit aee6a64 into go-mgo:v2-unstable Jul 14, 2016

babbageclunk added a commit to babbageclunk/juju that referenced this pull request Aug 18, 2016

Update mgo dependency

1248665

This version includes the fixes for the duplicate key error on upsert, which means we don't need the patch any more. go-mgo/mgo#291 go-mgo/mgo#316

babbageclunk mentioned this pull request Aug 18, 2016

Update mgo dependency juju/juju#6028

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Retry upserts that fail with "duplicate key error" #291

Retry upserts that fail with "duplicate key error" #291

babbageclunk commented Jun 27, 2016

niemeyer commented Jun 28, 2016

babbageclunk commented Jun 28, 2016

babbageclunk commented Jul 5, 2016

babbageclunk commented Jul 5, 2016

babbageclunk commented Jul 6, 2016

babbageclunk commented Jul 6, 2016

babbageclunk commented Jul 6, 2016

niemeyer Jul 8, 2016

babbageclunk Jul 10, 2016

babbageclunk commented Jul 10, 2016

babbageclunk commented Jul 10, 2016 •

edited

Loading

rogpeppe Jul 12, 2016

babbageclunk Jul 12, 2016

rogpeppe commented Jul 12, 2016

rogpeppe commented Jul 12, 2016 •

edited

Loading

jameinel Jul 14, 2016

babbageclunk Jul 14, 2016

niemeyer Jul 14, 2016

niemeyer commented Jul 14, 2016

babbageclunk commented Jul 14, 2016

babbageclunk commented Jul 14, 2016

Retry upserts that fail with "duplicate key error" #291

Retry upserts that fail with "duplicate key error" #291

Conversation

babbageclunk commented Jun 27, 2016

niemeyer commented Jun 28, 2016

babbageclunk commented Jun 28, 2016

babbageclunk commented Jul 5, 2016

babbageclunk commented Jul 5, 2016

babbageclunk commented Jul 6, 2016

babbageclunk commented Jul 6, 2016

babbageclunk commented Jul 6, 2016

niemeyer Jul 8, 2016

Choose a reason for hiding this comment

babbageclunk Jul 10, 2016

Choose a reason for hiding this comment

babbageclunk commented Jul 10, 2016

babbageclunk commented Jul 10, 2016 • edited Loading

rogpeppe Jul 12, 2016

Choose a reason for hiding this comment

babbageclunk Jul 12, 2016

Choose a reason for hiding this comment

rogpeppe commented Jul 12, 2016

rogpeppe commented Jul 12, 2016 • edited Loading

jameinel Jul 14, 2016

Choose a reason for hiding this comment

babbageclunk Jul 14, 2016

Choose a reason for hiding this comment

niemeyer Jul 14, 2016

Choose a reason for hiding this comment

niemeyer commented Jul 14, 2016

babbageclunk commented Jul 14, 2016

babbageclunk commented Jul 14, 2016

babbageclunk commented Jul 10, 2016 •

edited

Loading

rogpeppe commented Jul 12, 2016 •

edited

Loading