-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sql: TestLogic_upsert_non_metamorphic failed due to too large batch size #117070
Comments
Looks like this is a dupe of #116762, but that was closed by skipping 3-node-tenant under race. I was not able to reproduce this by running for an hour on a gceworker with:
|
I notice the mutations continue adding to a batch until the condition |
Removing release-blocker label since we believe this to be an unrealistic batch size setup. |
I haven't been able to reproduce, but I'm pretty sure the issue I mentioned here is what's causing the failure. The test uses rows of size ~100KB, and the failures show the limit exceeded by ~10KB. For the related copy test |
[triage] maybe we could make limit 5 MB instead of 4 in the test? |
The tests `TestLargeDynamicRows` and `TestLogic_upsert_non_metamorphic` occasionally flake because they set the raft command size limit to the minimum `4MiB`, and their batch size limiting is inexact. This commit prevents the flake by increasing the limit to `5MiB`. Making the batch size limit exact will still be tracked by cockroachdb#117070. Informs cockroachdb#117070 Release note: None
120419: sqlstats: simplify transaction latency test r=abarganier,xinhaoz a=dhartunian Remove need for test case counter which causes a data race. Fixes: #119580 Epic: None Release note: None 120633: physicalplan: bias towards streaks in bulk planning r=dt a=dt The bulk oracle is used when planning large, bulk jobs, that are expected to involve many or all ranges in a cluster, where all nodes are likely to be assigned a large number of spans, and overall plan and the specs that represent it will include a very large number of distinct spans. These large numbers of distinct spans in the specs can increase the cost of executing such a plan. In particular, processes that maintain a frontier of spans processes or not processed or time at which they are processed, such as CDC and PCR, have to track far more distinct spans in large clusters. We can, however, in some cases reduce this number of distinct spans, by biasing the assignment of key ranges to nodes during replica selection to pick the same node for sequential ranges. By assigning, say, 10 spans to node one, then ten to two, then ten to three, potentially each node is now only tracking one logical span, that is 10x wider, instead of ten distinct spans. We can bias towards such streaks only when the candidate replicas for a span include one on the node that would extend the streak, so this is an opportunistic optimization that depends on replica placement making it an option. Additionally we need to be careful when applying such a bias that we still *distribute* work roughly evenly to achieve our desired overall utilization of the cluster. Thus we only bias towards streaks when the streak length is short or when the node on which we are extending a streak remains within some multiple of the least assigned node, reverting to the normal random selection if this is not the case. Release note: none. Epic: none. 120766: sql: increase raft command size limit for some tests r=DrewKimball a=DrewKimball The tests `TestLargeDynamicRows` and `TestLogic_upsert_non_metamorphic` occasionally flake because they set the raft command size limit to the minimum `4MiB`, and their batch size limiting is inexact. This commit prevents the flake by increasing the limit to `5MiB`. Making the batch size limit exact will still be tracked by #117070. Informs #117070 Release note: None 120769: sqlstats: skip TestSQLStatsCompactor r=abarganier a=dhartunian Release note: None 120781: streamingest: skip `TestStreamingReplanOnLag` r=rail a=rickystewart This test is very flaky. See #120688 Epic: none Release note: None Co-authored-by: David Hartunian <davidh@cockroachlabs.com> Co-authored-by: David Taylor <tinystatemachine@gmail.com> Co-authored-by: Drew Kimball <drewk@cockroachlabs.com> Co-authored-by: Ricky Stewart <ricky@cockroachlabs.com>
This keeps failing, so increasing the priority to P-2 |
We need to:
|
It looks like the roachtest failures are using the default command size limit, so the same fix might not apply. |
I'm lowering this to P-3 since we haven't seen this failure in logic tests in a while and moving it to the backlog. Note that there is a separate issue for the COPY roachtest #121413 which seems more frequent, so that one is still P-2. |
pkg/sql/logictest/tests/fakedist-vec-off/fakedist-vec-off_test.TestLogic_upsert_non_metamorphic failed on master @ 53e4efd389167110d687e5dea8bf915fe2582a25:
Parameters:
attempt=1
race=true
run=1
shard=41
Help
See also: How To Investigate a Go Test Failure (internal)
This test on roachdash | Improve this report!
Jira issue: CRDB-34955
The text was updated successfully, but these errors were encountered: