[YSQL] ON CONFLICT (k) DO UPDATE can still throw conflict errors

[aphyr]

With version 1.3.1.0, using Jepsen ed8ecfb9146816cb99e458c9cb92f59e19a7b78f, the append test uses insert ... on conflict (k) do update ... to insert or update records in place. This table has two columns: a primary integer key k, and a text v.

           :append
           (do (c/execute! conn
                           [(str "insert into " table " (k, v) values (?, ?) "
                                 "on conflict (k) do update set "
                                 "v = CONCAT(" table ".v, ',', ?)")
                            k (str v) (str v)])

You would think that when there exists a row with some key k, this would update the associated v using CONCAT, and indeed it does--some of the time. Other times, however, it throws:

ERROR: duplicate key value violates unique constraint "append1_pkey"

See, for instance, 20190806T095424.000-0400.zip

This happens whether we use on conflict (k) or explicitly specify the index viaon conflict on constraint append1_pkey", and it happens regardless of whether k is a primary key, or simply a unique column. You can reproduce it with any append test, e.g.

lein run test --os debian --version 1.3.1.0 --time-limit 300 -w ysql/append --concurrency 5n

[aphyr]

This might be related to #1611!

[frozenspider]

Re-tested this under @nocaway 's recent partial fix for #1611 (commit ca3ebdb), this bug is still reproducible.
Logs: 20190807T154851.000Z.zip

[frozenspider]

@nocaway discovered this bug to be unrelated to #1611
From his description:

The errors occurred whenever one or all of the following conditions occurred.

• Client was disconnected and reconnected a few times (Server crashes?)
• Transaction layer sends out two kinds of error messages
• Error log also report transaction error.

ERROR log has

E0806 14:00:04.040017 47479 transaction_participant.cc:791] T b08b3de1c71a460088dc3f13408e796c P 271c085f0ca84a0d9ff3cbaa4c593f18: Transaction should not have metadata but present: b82c901b-a719-40c1-8119-a2cbab21a465

INFO log has transaction info

2019-08-06 13:57:20.058 UTC [3428] ERROR:  Operation failed. Try again.: Conflicts with higher priority transaction: 51539a97-818b-4369-8a06-9146b8915009
W0806 13:58:35.234104 16176 transaction.cc:747] 97ee01e9-6b88-4eb4-86b8-384b2a7896b9: Send heartbeat failed: Operation expired (yb/tablet/transaction_coordinator.cc:794): Transaction expired