Add a FAQ on contention #3156

knz · 2018-05-17T11:35:53Z

I've been seeing this so many times everywhere (on the forum, on gitter, from sales etc) that I am surprised we didn't FAQ this earlier. Here it is.

cockroach-teamcity · 2018-05-17T11:35:59Z

This change is

knz · 2018-05-17T11:36:11Z

Once the copy-editing is done I'll do the propagation to the other version directories

cockroach-teamcity · 2018-05-17T11:38:43Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/bf002dc13a534c04dccb2c5ea3a2223f8c85ec08/

cockroach-teamcity · 2018-05-17T11:40:19Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/4a8ee438a663807f7a781f5aa80a7399034173fd/

cockroach-teamcity · 2018-05-17T11:52:18Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/a90321916660dcb0f65a71d276625c0754c980be/

cockroach-teamcity · 2018-05-17T11:57:29Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/80362761a5f15be7a314c9770715d56e6bd2e145/

benesch · 2018-05-17T12:11:24Z

v2.1/sql-faqs.md

+  over table rows with the same index key values (either on [primary
+  keys](primary-key.html) or secondary [indexes](indexes.html)) or using
+  index key values that are close to each other, and thus place the
+  indexed data on the same [data ranges](architecture/overview.html).


table rows with the same index key values within the same column family, no?a

That's what I wrote first. But for identifying contention column families don't matter: we don't split rows with the same PK across different ranges, so even accesses to different column families land on the same range.

The distinction matters for single-key serialization, which I detail further below.

cockroach-teamcity · 2018-05-17T12:13:10Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/8915bc846db781d2a3fcc39fbcb216e8209a1a06/

cockroach-teamcity · 2018-05-17T12:18:35Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/76395ea977ae3d3aed625f34293c5f8e795455df/

cockroach-teamcity · 2018-05-17T12:19:51Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/9fd8a3f7c9cd8e511aa7888a3a09de8f02c43d60/

bdarnell · 2018-05-17T17:42:31Z

Does this belong in FAQs or the performance best practices page? It's definitely a frequently-encountered issue, but "what is transaction contention" is not the way that people phrase the question. Is there another title we can give this that would be a better match for the users' language? (other than something vague like "why is it so slow")

Review status: 0 of 1 files reviewed at latest revision, 1 unresolved discussion, all commit checks successful.

v2.1/sql-faqs.md, line 58 at r2 (raw file):

- At least some of the transactions write or modify the data.

Transaction contention will cause contended transactions to be

"A set of transactions that all contend on the same keys will be limited in performance to the maximum processing speed of a single node..."

It is possible for a set of transactions that all contend with each other to contend on different keys and therefore still get some parallelism, but we don't need to go into that much detail here.

v2.1/sql-faqs.md, line 69 at r2 (raw file):

  node (the range lease holder). Performance can increase if CockroachDB
  can utilize multiple hardware processors, i.e. some horizontal
  scalability via multi-core parallelism is possible.

I would remove the discussion of multi-core parallelism from these two bullets, and instead add a section at the end, something like:

"It is always best to avoid contention as much as possible via the design of the schema and application. However, sometimes contention is unavoidable. To maximize performance in the presence of contention, you'll need to maximize the performance of a single range.

Minimize the network distance between the replicas of a range, possibly using zone configs and partitioning.
Use the fastest storage devices available.
If the contending transactions operate on different keys within the same range, add more CPU power (more cores) per node (this is less likely to provide an improvement if the transactions all operate on the same key)."

v2.1/sql-faqs.md, line 90 at r2 (raw file):

  suggestions.

- Increase

I'm not sure about this advice. Have we seen many instances where this was part of the solution? Denormalization of immutable (or nearly-immutable) data can also increase scalability by creating multiple copies of often-referenced data.

v2.1/sql-faqs.md, line 106 at r2 (raw file):

  [`SELECT`](select-clause.html) and
  [`INSERT`](insert.html)/[`UPDATE`](update.html)/[`DELETE`](delete.html)/[`UPSERT`](upsert.html)
  clauses together in a single SQL statement.

Also mention that using UPSERT and specifying values for all columns will usually have the best performance under contention (compared to INSERT or UPDATE).

Comments from Reviewable

knz · 2018-05-17T17:51:44Z

The title must stay because we often talk about contention in our explanations to users (and in blog posts, and other things) and it would be good for people to have a definition/explanation to search for if they want to understand our answers better.

However I agree we must provide additional entry points to make this information better discoverable. I'll make some attempt. Will let you know.

Review status: 0 of 1 files reviewed at latest revision, 5 unresolved discussions, all commit checks successful.

Comments from Reviewable

knz · 2018-05-17T18:24:05Z

I have moved the bulk of the explanation to the perf best practices page, and added two entry points in the FAQ.

RFAL

Review status: 0 of 2 files reviewed at latest revision, 5 unresolved discussions.

v2.1/sql-faqs.md, line 58 at r2 (raw file):

Previously, bdarnell (Ben Darnell) wrote…

"A set of transactions that all contend on the same keys will be limited in performance to the maximum processing speed of a single node..."

It is possible for a set of transactions that all contend with each other to contend on different keys and therefore still get some parallelism, but we don't need to go into that much detail here.

Done.

v2.1/sql-faqs.md, line 69 at r2 (raw file):

Previously, bdarnell (Ben Darnell) wrote…

I would remove the discussion of multi-core parallelism from these two bullets, and instead add a section at the end, something like:

"It is always best to avoid contention as much as possible via the design of the schema and application. However, sometimes contention is unavoidable. To maximize performance in the presence of contention, you'll need to maximize the performance of a single range.

Minimize the network distance between the replicas of a range, possibly using zone configs and partitioning.

Use the fastest storage devices available.

If the contending transactions operate on different keys within the same range, add more CPU power (more cores) per node (this is less likely to provide an improvement if the transactions all operate on the same key)."

Done.

v2.1/sql-faqs.md, line 90 at r2 (raw file):

Previously, bdarnell (Ben Darnell) wrote…

I'm not sure about this advice. Have we seen many instances where this was part of the solution? Denormalization of immutable (or nearly-immutable) data can also increase scalability by creating multiple copies of often-referenced data.

Added a note with your highlight.

v2.1/sql-faqs.md, line 106 at r2 (raw file):

Previously, bdarnell (Ben Darnell) wrote…

Also mention that using UPSERT and specifying values for all columns will usually have the best performance under contention (compared to INSERT or UPDATE).

Done.

Comments from Reviewable

cockroach-teamcity · 2018-05-17T18:25:55Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/85f87a9fc7b8724e7b7886cef39d74565b074f1e/

cockroach-teamcity · 2018-05-17T18:28:17Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/1f112e524e3365c3dc5e67c780c93ac22b1f91b8/

bdarnell · 2018-05-21T01:30:03Z

Review status: 0 of 2 files reviewed at latest revision, 5 unresolved discussions, all commit checks successful.

Comments from Reviewable

jseldess

This is awesome, @knz! As usual. LGTM, with some nits.

Once this lands, I'll look into other ways to cross-reference this new material.

jseldess · 2018-05-22T13:51:21Z

v2.1/performance-best-practices-overview.md

+
+## Understanding and Avoiding Transaction Contention
+
+Transaction contention occurs when the following three conditions hold


nit: hold together > are met.

jseldess · 2018-05-22T13:52:36Z

v2.1/performance-best-practices-overview.md

+
+To avoid contention, multiple strategies can be applied:
+
+- Use index key values with a more random distribution of values, so


Probably best to link directly to the specific FAQ you have in mind.

jseldess · 2018-05-22T13:53:46Z

v2.1/performance-best-practices-overview.md

+  specify values for all columns in the inserted rows. This will
+  usually have the best performance under contention, compared to
+  combinations of [`SELECT`](select-clause.html),
+  [`INSERT`](insert.html) and [`UPDATE`](update.html).


nit: place comma before the and (the "oxford comma").

jseldess · 2018-05-22T13:55:42Z

v2.1/performance-best-practices-overview.md

+It is always best to avoid contention as much as possible via the
+design of the schema and application. However, sometimes contention is
+unavoidable. To maximize performance in the presence of contention,
+you'll need to maximize the performance of a single range.


Is this paragraph introducing the following bulleted list? If so, use a semicolon instead of a period at the end of the last sentence.

jseldess · 2018-05-22T13:56:33Z

v2.1/sql-faqs.md

+Transaction contention occurs when transactions issued from multiple
+clients at the same time operate on the same data.
+
+This can cause transactions to wait on each other and decrease


This doesn't need to be its own paragraph. Please make it the second sentence of the previous paragraph.

jseldess · 2018-05-22T13:57:14Z

v2.1/sql-faqs.md

+Transaction
+Contention](performance-best-practices-overview.html#understanding-and-avoiding-transaction-contention).
+
+## Why can I not get more operations per second by increasing the number of nodes?


I'd rephrase this a bit: Why would increasing the number of nodes not result in more operations per second?

knz · 2018-05-22T14:12:09Z

Review status: 0 of 2 files reviewed at latest revision, 11 unresolved discussions.

v2.1/performance-best-practices-overview.md, line 232 at r3 (raw file):