New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unclear documentation for “transaction is too large to complete; try splitting into pieces” #4204

Open
sploiselle opened this Issue Dec 19, 2018 · 1 comment

Comments

Projects
None yet
2 participants
@sploiselle
Copy link
Collaborator

sploiselle commented Dec 19, 2018

From the forum:

I’m running in to the transaction is too large to complete; try splitting into pieces error. I initially tried to fix the problem as suggested in the documentation by breaking a large INSERT INTO ... SELECT FROM statement into multiple statements inside a single transaction. However, this doesn’t resolve the issue and I still get the same error even if each statement is only a few hundred bytes. After reading the documentation more carefully I noticed an important detail: “manually page through the data you want to insert using separate transactions” (emphasis mine).

So it sounds like I have to break apart the transaction into multiple transactions rather than break it up into multiple statements within the transaction. This should probably be spelled out much more clearly in the documentation because this is the type of error that you would only encounter in the wild after deploying something. I only found it after running some load tests with very large input data.

This is complicated further by the claims made in the same section which the average reader would incorrectly assume applied to this problem:

“A single statement can perform at most 64MiB of combined updates”
and
“There’s only a size limit, and it applies independently to each statement”

https://www.cockroachlabs.com/docs/stable/known-limitations.html#write-and-update-limits-for-a-single-statement

I’m enjoying using CockroachDB but the size limitations are challenging to deal with.

Possibly also a good use case for IMPORT, which would be a better recommendation from the CLI than the current error in this circumstance.

@rmloveland

This comment has been minimized.

Copy link
Contributor

rmloveland commented Dec 20, 2018

Additional information about this error from the engineering team:

That error means that the transaction has encountered a conflict with another transaction that will force it to restart.
(If it did not have to restart, it would be allowed to complete even if it exceeded the limit.)
So even if the limit were increased, the transaction would have to restart, and would be likely to be stuck in an endless cycle of restarts.
In other words, the transaction has to be made smaller to reduce the chance and cost of restarts, not because it exceeds this particular limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment