etcd: STM transaction queue to effectively reduce retries for conflicting transactions #4457

bhandras · 2020-07-10T13:43:26Z

~~rebased on #4411~~
now on master since #4411 is merged

This PR adds (and integrates) commitQueue which' purpose is to detect conflicts for concurrently applied transactions and effectively reduce retries, by applying queuing up conflicting transactions for sequential execution, while leaving all non-conflicting ones to run freely (potentially in parallel).

Roasbeef

The implementation is much simpler than I thought it would be! Just completed an initial pass, and nothing glaring jumped out. Will do another pass once I run it on an actual replicated db lnd instance. It would also be interesting to create a small patch that lets us run certain itests w/ and w/o this change so we can gauge the rough impact of the change on perf.

channeldb/kvdb/etcd/commit_queue.go

Roasbeef · 2020-08-14T02:32:44Z

channeldb/kvdb/etcd/commit_queue.go

+		if !blocked {
+			_, rsetContainsKey := rset[key]
+			blocked = (c.writerMap[key] > 1 ||
+				(c.readerMap[key] > 0 && !rsetContainsKey))


Why don't we need to block if there's a pending transaction in the queue that reads this key, we want to write it, but don't also read the key ourselves?

It's because if our read set contains the key then we already increased c.readerMap[key] above, so to make sure reader lock count is non zero we have to "uncount ourselves".

I think this would be much easier to reason about by just by doing two passes through the sets:

for key := range rset { blocked |= c.writerMap[key] > 0 } for key := range wset { blocked |= c.writerMap[key] > 0 || c.readerMap[key] > 0 } for key := range rset { c.readerMap[key] += 1 } for key := range wset { c.writerMap[key] += 1 }

Performance wise I doubt we'll see any difference.

Yeah, it's a bit hard to read... Unfortunately we can't use the the simplified version above because if the same transaction also reads the key (where no other readers are present) then will unnecessary block. This is the reason for the rsetContainsKey variable. Added a few comments to clarify. I'm open to any suggestions you may find that simplifies though.

** and also simplified a bit to make it more? readable

channeldb/kvdb/etcd/db.go

cfromknecht

very cool how small the diff is!

channeldb/kvdb/etcd/db.go

channeldb/kvdb/etcd/embed.go

channeldb/kvdb/etcd/commit_queue.go

cfromknecht · 2020-08-18T02:57:32Z

channeldb/kvdb/etcd/stm.go

+	// Run the tx closure to construct the read and write sets.
+	// Also we expect that if there are no conflicting transactions
+	// in the queue, then we only run apply once.
+	if err = apply(s); err != nil {


is it intentional that this shadows the err in the outer scope? o/w i don't see where that error is read?

Yes, so it's a bit tricky to read this at first, but it's really simple actually.

What we do is we first run the apply closure to gather the read/write sets so we can add the tx to the contention queue.

The execute closure is executed there (either immediately or in the queue goroutine).

The err simply holds the error trough the above described execution graph.
3) we wait for the done signal and them clean the keys from the queue.

thanks for the explanation, makes sense now! i also see that it's the return value at the end of the function, so that's where it is "read"

Since we return immediately if the error is non-nil here, wouldn't if err := apply(s); err != nil be equivalent?

I agree that the shadowing is tricky to read. Could make sense to add more errors with descriptive names (i.e. executeErr) to make it easier.

Yeah, maybe it's simpler to read if we distinguish errors by scope. PTAL

channeldb/kvdb/etcd/commit_queue.go

channeldb/kvdb/etcd/commit_queue_test.go

bhandras · 2020-08-28T14:44:10Z

The implementation is much simpler than I thought it would be! Just completed an initial pass, and nothing glaring jumped out. Will do another pass once I run it on an actual replicated db lnd instance. It would also be interesting to create a small patch that lets us run certain itests w/ and w/o this change so we can gauge the rough impact of the change on perf.

Yes, originally the queue was optional but decided to make it non-optional as really it should be on all the time.
We can still do the comparison, as it just requires removing these commits from the itest PR (#4402)

cfromknecht

LGTM 🌮

halseth

Fun change :)

halseth · 2020-09-16T07:51:10Z

channeldb/kvdb/etcd/commit_queue.go

+	for key := range rset {
+		c.readerMap[key] += 1
+		if !blocked {
+			blocked = c.writerMap[key] > 0


style suggestion: blocked ||= c.writerMap[key] > 0

halseth · 2020-09-16T10:10:29Z

channeldb/kvdb/etcd/stm.go

+	// Run the tx closure to construct the read and write sets.
+	// Also we expect that if there are no conflicting transactions
+	// in the queue, then we only run apply once.
+	if err = apply(s); err != nil {


Since we return immediately if the error is non-nil here, wouldn't if err := apply(s); err != nil be equivalent?

I agree that the shadowing is tricky to read. Could make sense to add more errors with descriptive names (i.e. executeErr) to make it easier.

halseth · 2020-09-16T10:21:14Z

channeldb/kvdb/etcd/commit_queue.go

+
+// Wait waits for the queue to stop (after the queue context has been canceled).
+func (c *commitQueue) Wait() {
+	<-c.done


use the more common waitgroup pattern instead?

halseth · 2020-09-16T10:26:55Z

channeldb/kvdb/etcd/commit_queue.go

+		if !blocked {
+			_, rsetContainsKey := rset[key]
+			blocked = (c.writerMap[key] > 1 ||
+				(c.readerMap[key] > 0 && !rsetContainsKey))


I think this would be much easier to reason about by just by doing two passes through the sets:

for key := range rset { blocked |= c.writerMap[key] > 0 } for key := range wset { blocked |= c.writerMap[key] > 0 || c.readerMap[key] > 0 } for key := range rset { c.readerMap[key] += 1 } for key := range wset { c.writerMap[key] += 1 }

Performance wise I doubt we'll see any difference.

channeldb/kvdb/etcd/commit_queue.go

bhandras

Thanks for the review @halseth! Main change is the (hopefully) more readable rset/wset scans. PTAL

bhandras · 2020-09-16T11:21:04Z

channeldb/kvdb/etcd/commit_queue.go

+
+// Wait waits for the queue to stop (after the queue context has been canceled).
+func (c *commitQueue) Wait() {
+	<-c.done


bhandras · 2020-09-16T11:48:56Z

channeldb/kvdb/etcd/commit_queue.go

+		if !blocked {
+			_, rsetContainsKey := rset[key]
+			blocked = (c.writerMap[key] > 1 ||
+				(c.readerMap[key] > 0 && !rsetContainsKey))


Yeah, it's a bit hard to read... Unfortunately we can't use the the simplified version above because if the same transaction also reads the key (where no other readers are present) then will unnecessary block. This is the reason for the rsetContainsKey variable. Added a few comments to clarify. I'm open to any suggestions you may find that simplifies though.

bhandras · 2020-09-16T11:49:01Z

channeldb/kvdb/etcd/commit_queue.go

+	for key := range rset {
+		c.readerMap[key] += 1
+		if !blocked {
+			blocked = c.writerMap[key] > 0


channeldb/kvdb/etcd/commit_queue.go

bhandras · 2020-09-16T12:02:49Z

channeldb/kvdb/etcd/stm.go

+	// Run the tx closure to construct the read and write sets.
+	// Also we expect that if there are no conflicting transactions
+	// in the queue, then we only run apply once.
+	if err = apply(s); err != nil {


Yeah, maybe it's simpler to read if we distinguish errors by scope. PTAL

halseth · 2020-09-16T12:38:43Z

channeldb/kvdb/etcd/commit_queue.go

+			// Transaction is blocked if:
+			// - there's any reader (which is not this tx).
+			// - there's any writer.
+			blocked = blocked || (c.readerMap[key] > 0 && !keyRead)


still not sure if this is correct. Say this tx reads and writes this key, increases c.readerMap[key] to 2.

That will leave (c.readerMap[key] > 0 && !keyRead) == false while it should be blocked.

That was a really nice catch!

After some back and forth, decided to go with the simplified version above just with three loops.

Roasbeef

I found the latest iteration much easier to reason about this time around, kudos to the prior reviewers in this series!

LGTM 🚁

Should wait to merge this till we get 3/3 since the last iteration had a nice find.

channeldb/kvdb/etcd/commit_queue.go

Roasbeef · 2020-09-16T23:54:13Z

.github/workflows/main.yml

@@ -214,6 +214,7 @@ jobs:
      matrix:
        unit_type:
          - btcd unit-cover
+          - unit tags=kvdb_etcd


halseth

LGTM now, great work! 😀

halseth · 2020-09-17T11:15:37Z

channeldb/kvdb/etcd/stm.go

+	// Run the tx closure to construct the read and write sets.
+	// Also we expect that if there are no conflicting transactions
+	// in the queue, then we only run apply once.
+	if preApplyErr := apply(s); preApplyErr != nil {


halseth · 2020-09-17T11:16:56Z

channeldb/kvdb/etcd/commit_queue.go

+	// the read set. Do not increment the reader counts yet as we'll need to
+	// use the original read counts when scanning through the write set.
+	for key := range rset {
+		blocked = blocked || c.writerMap[key] > 0


nit: optimization here and below, can immediately break loop if already blocked.

This commit adds commitQueue which is a lightweight contention manager for STM transactions. The queue attempts to queue up transactions that conflict for sequential execution, while leaving all "unblocked" transactons to run freely in parallel.

This commit integrates an externally passed commitQueue instance with the STM to reduce retries for conflicting transactions.

bhandras · 2020-09-17T12:51:10Z

Thanks everyone for the reviews!

bhandras force-pushed the etcd_tx_queue branch from 32d7116 to 3ce8bc5 Compare July 10, 2020 13:44

bhandras mentioned this pull request Jul 10, 2020

itests: option to run our integration tests on etcd + boltdb (remote/local) #4402

Merged

bhandras requested review from cfromknecht and Roasbeef July 13, 2020 08:14

Roasbeef added database Related to the database/storage of LND etcd optimization labels Jul 20, 2020

bhandras force-pushed the etcd_tx_queue branch 3 times, most recently from 6e5b085 to e8fb359 Compare August 10, 2020 14:52

bhandras changed the title ~~wip tx queue~~ STM transaction queue to effectively reduce retries for conflicting transactions Aug 10, 2020

bhandras force-pushed the etcd_tx_queue branch from e8fb359 to e63657f Compare August 10, 2020 14:57

bhandras marked this pull request as ready for review August 10, 2020 14:59

bhandras changed the title ~~STM transaction queue to effectively reduce retries for conflicting transactions~~ etcd: STM transaction queue to effectively reduce retries for conflicting transactions Aug 10, 2020

bhandras added this to the 0.12.0 milestone Aug 10, 2020

bhandras added this to In progress in v0.12.0-beta via automation Aug 10, 2020

Roasbeef moved this from In progress to Review in progress in v0.12.0-beta Aug 12, 2020

Roasbeef reviewed Aug 14, 2020

View reviewed changes

cfromknecht reviewed Aug 18, 2020

View reviewed changes

bhandras force-pushed the etcd_tx_queue branch from e63657f to 1862eef Compare August 28, 2020 14:40

bhandras requested review from Roasbeef and cfromknecht August 28, 2020 14:41

bhandras force-pushed the etcd_tx_queue branch 3 times, most recently from 0cb3658 to cf25382 Compare September 4, 2020 14:20

cfromknecht approved these changes Sep 11, 2020

View reviewed changes

bhandras requested a review from halseth September 16, 2020 06:37

halseth suggested changes Sep 16, 2020

View reviewed changes

bhandras commented Sep 16, 2020

View reviewed changes

bhandras requested a review from halseth September 16, 2020 12:05

bhandras force-pushed the etcd_tx_queue branch from cf25382 to 6efdf97 Compare September 16, 2020 12:07

halseth reviewed Sep 16, 2020

View reviewed changes

bhandras requested a review from halseth September 16, 2020 12:41

bhandras force-pushed the etcd_tx_queue branch 2 times, most recently from aec925f to 8ad8dcd Compare September 16, 2020 13:26

Roasbeef approved these changes Sep 16, 2020

View reviewed changes

halseth approved these changes Sep 17, 2020

View reviewed changes

v0.12.0-beta automation moved this from Review in progress to Reviewer approved Sep 17, 2020

bhandras added 6 commits September 17, 2020 14:50

etcd: increase message and transaction limits for embedded etcd

b4b5a9d

etcd: make embedded etcd context cancelable

6f3a45b

etcd: integrate the commitQueue to the STM commit loop

9c47392

This commit integrates an externally passed commitQueue instance with the STM to reduce retries for conflicting transactions.

make: allow optional extra tags when running unit tests

357cd7d

build: unit test on github with kvdb_etcd tag

26effca

bhandras force-pushed the etcd_tx_queue branch from 8ad8dcd to 26effca Compare September 17, 2020 12:50

bhandras merged commit 111db80 into lightningnetwork:master Sep 17, 2020

v0.12.0-beta automation moved this from Reviewer approved to Done Sep 17, 2020

bhandras deleted the etcd_tx_queue branch September 12, 2023 15:28

etcd: STM transaction queue to effectively reduce retries for conflicting transactions #4457

etcd: STM transaction queue to effectively reduce retries for conflicting transactions #4457

Conversation

bhandras commented Jul 10, 2020 • edited

Roasbeef left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cfromknecht left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bhandras commented Aug 28, 2020

cfromknecht left a comment

Choose a reason for hiding this comment

halseth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bhandras left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Roasbeef left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

halseth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bhandras commented Sep 17, 2020

bhandras commented Jul 10, 2020 •

edited