Docs inserting with recent_sequences=0 in syncgateway when one of couchbase node goes down

[ArihantRk]

I have syncgateway configured to couchbase cluster of 2 nodes. i could see that documents are inserting with "recent_sequences": [0],while inserting documents to couchbase when one of couchbase node went down.

in this time i could some documents are failed with error code as 500 due to one node down. but other documents which are inserted to active node with below format. Also these documents will not be access from clients.

{
"_attachments": {
"message.txt": {
"content_type": "text/plain",
"digest": "sha1-CmaZ2nH43QkNZHriiqA+MRZNyj0=",
"length": 22,
"revpos": 1,
"stub": true
}
},
"_sync": {
"rev": "1-294911935a141ae695db2b1addb83af4",
"recent_sequences": [
0
],

below are the doughts i have with this behaviour
1.what is the significance of creating docs with like this
2. whenever one CB node goes down, whatever docs inserted between this time, no docs inserted properly in couchbase
3. is this happens when _sync:seq documents exists in down node?

[adamcfraser]

The recent_sequences are less important than the actual sequence number on the document - is there a sequence value in the _sync property for your document?

Also, what version of Sync Gateway are you running?

[ArihantRk]

@adamcfraser , there is no actual sequence number in doc. am running syncgateway 1.2.1 version.

[adamcfraser]

@ArihantRk Can you post your full Sync Gateway logs to a gist when you hit this issue? I'm assuming it's related to what you mention in #3 - the vbucket associated with the _sync:seq document is assigned to the node that is down - but in that scenario the incr operation to generate a new sequence should fail, and the document write should fail.

[ArihantRk]

@adamcfraser , please find the gist log in below https://gist.github.com/ArihantRk/237e4c2eb1451e97e97bff58fb034c32

i could some documents are failed with return code 500 which are related to down node.

[adamcfraser]

@ArihantRk Which documents from that log were successfully written, but with a missing sequence?

[ArihantRk]

you can check for ids 617,641 and 642 which have missing sequence

[adamcfraser]

@ArihantRk Thanks for filing this report - there's a bug in the incr retry processing. I'll get a fix pushed to master as soon as possible, and include in the upcoming 1.3 release.

[adamcfraser]

@ArihantRk I've merged what should be the fix for this to master - if you're able to retest with a pinned dependency build from master, it would be helpful to get feedback. There's details on how to build here: https://github.com/couchbase/sync_gateway/blob/master/README.md

If not, I've filed couchbaselabs/mobile-testkit#496 to reproduce your issue with a functional test against a multi-node cluster - you can watch that for confirmation of the fix.

[ArihantRk]

Thanks @adamcfraser , i will try .

[ArihantRk]

Thanks @adamcfraser . I have tested with latest build. I had not seen issue .