Upsert transactions may upsert multiple copies of a record

[aphyr]

The howto says

Upsert operations are intended to be run concurrently, as per the needs of the application. As such, it’s possible that two concurrently running operations could try to add the same node at the same time. For example, both try to add a user with the same email address. If they do, then one of the transactions will fail with an error indicating that the transaction was aborted.

However, multiple concurrent upsert transactions for the same key may succeed, resulting in the creation of multiple, rather than one, records for that key. For instance, in 20180216T075315.000-0600.zip, four upserts complete on the same key, resulting in entities 0x01, 0x02, 0x04, and 0x05:

3	:invoke	:upsert	nil
0	:invoke	:upsert	nil
2	:invoke	:upsert	nil
4	:invoke	:upsert	nil
1	:invoke	:upsert	nil
3	:ok	:upsert	"0x1"
4	:ok	:upsert	"0x5"
1	:ok	:upsert	"0x2"
0	:ok	:upsert	"0x4"
2	:fail	:upsert	nil	:conflict
1	:invoke	:read	nil
4	:invoke	:read	nil
3	:invoke	:read	nil
2	:invoke	:read	nil
0	:invoke	:read	nil
4	:ok	:read	("0x1" "0x2" "0x4" "0x5")
2	:ok	:read	("0x1" "0x2" "0x4" "0x5")
3	:ok	:read	("0x1" "0x2" "0x4" "0x5")
1	:ok	:read	("0x1" "0x2" "0x4" "0x5")
0	:ok	:read	("0x1" "0x2" "0x4" "0x5")

This is easy to reproduce on a fresh cluster with no failures. You can demonstrate this failure mode with Jepsen 07abf364c2364c710f08a6f49392851a94f83c76, on 1.0.3, with lein run test -w upsert.

[aphyr]

(This is just for completeness; this issue was fixed by 7f9c659)

[aphyr]

(This is just for completeness; this issue was fixed by 7f9c659)

[aphyr]

I think I may have spoke too soon; this also looks to be present on

Dgraph version   : v1.0.4
Commit SHA-1     : 224b560
Commit timestamp : 2018-03-27 16:56:17 +0530
Branch           : fix/jepsen_delete

For instance, 20180330T110412.000-0500.zip shows 10/10 processes successfully upserting to a single key, where only one should succeed.

[aphyr]

I think I may have spoke too soon; this also looks to be present on

Dgraph version   : v1.0.4
Commit SHA-1     : 224b560
Commit timestamp : 2018-03-27 16:56:17 +0530
Branch           : fix/jepsen_delete

For instance, 20180330T110412.000-0500.zip shows 10/10 processes successfully upserting to a single key, where only one should succeed.

[janardhan1993]

@aphyr There was a change in v1.0.4, we need to specify @upsert in the schema otherwise index keys won't be used for conflict detection.

[janardhan1993]

@aphyr There was a change in v1.0.4, we need to specify @upsert in the schema otherwise index keys won't be used for conflict detection.

[janardhan1993]

Fixed.

[janardhan1993]

Fixed.

[aphyr]

Yeah, looks like the dev build you gave me earlier passes, with the upsert directive. I've also expanded the upsert test to try hundreds of keys (instead of one), and with several hours of inserts with and without partitions, I can't reproduce the conflict any more. :)

Dgraph version : v1.0.4
Commit SHA-1 : 224b560
Commit timestamp : 2018-03-27 16:56:17 +0530
Branch : fix/jepsen_delete

[aphyr]

Yeah, looks like the dev build you gave me earlier passes, with the upsert directive. I've also expanded the upsert test to try hundreds of keys (instead of one), and with several hours of inserts with and without partitions, I can't reproduce the conflict any more. :)

Dgraph version : v1.0.4
Commit SHA-1 : 224b560
Commit timestamp : 2018-03-27 16:56:17 +0530
Branch : fix/jepsen_delete

[manishrjain]

Whoop! Whoop!

[manishrjain]

Whoop! Whoop!