roachtest: update tpcc overload

[andrewbaptist]

Previously this test was created but wasn't run since it would always fail. The test still doesn't pass in the default configuration, however after the addition of #118781, at least it is possible to configure a cluster to pass.

Informs: #110272
Informs: #89142

Release note: None

[cockroach-teamcity]

This change is 

[andrewbaptist]

Run with this PR server.max_open_transactions_per_gateway = 100

I'm running again without that parameter to see the difference.

[andrewbaptist]

This is the default behavior (without the parameter set). I "unset" the parameter at 15:35. Note that the throughput without the parameter is ~1/2 of before, and the latency is >100x.

[rafiss]

i think this is great, and nice experiment!

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @renatolabs and @srosenberg)

[andrewbaptist]

TFTR!

bors r=rafiss

[craig]

Build succeeded:

• examples_orms
• macos_amd64_build
• lint
• local_roachtest_fips
• Bazel Essential CI (Cockroach)
• linux_amd64_build
• linux_arm64_build
• acceptance
• local_roachtest
• windows_build
• docker_image_amd64
• check_generated_code
• macos_arm64_build
• linux_amd64_fips_build

[sumeerbhola]

Thanks for the test. I am running this now. This runs hourly full backups and 15 min incremental backups, which also consume significant resources. Was that intentional?

Also, what is the motivation for the 4h ramp? I was planning to speed it up to 30m to get to the overload quicker.

[sumeerbhola]

There is some uneven load distribution. n4 and n2 have higher CPU and higher KV read and write requests.

[andrewbaptist]

The reason for the 4h ramp is to allow a more gradual change from "sustainable to unsustainable" and watch the impact as it changes. The backups were not really intentional, but when I noticed them I didn't remove them since in the customer case they were somewhat impacted by backups. It would probably be cleaner to remove them.

From a rebalance perspective, it would be good to better understand if they converge over time and if not why not.

[sumeerbhola]

My run flaked out after about 75 min of the tpcc workload running due to. This was with server.max_open_transactions_per_gateway = -1.

_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
 4617.0s        0           48.9           33.6    436.2   1409.3   1543.5   1543.5 delivery
 4617.0s        0          425.5          334.8   1543.5   2147.5   2415.9   2550.1 newOrder
 4617.0s        0           94.9           33.6    104.9   1006.6   1208.0   1208.0 orderStatus
run_132808.222807000_n7_cockroach-workload-r: 14:47:20 cluster.go:2382: > result: _potential_ SSH flake (`ssh -vvv` log retained in artifacts/admission-control/tpcc-severe-overload/run_1/ssh/ssh_132808.262145000_n7_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255

Throughput was looking ok