tenantcostclient: figure out how to "unskip" TestEstimateQueryRUConsumption

[yuzefovich]

Currently, TestEstimateQueryRUConsumption is unconditionally skipped which is no longer allowed under "test vision zero". The test has been skipped since its introduction in #89256 with the following comment:

	// This test becomes flaky when the machine/cluster is under significant
	// background load, so it should only be run manually.

We should figure out how to enable this test at least in some cases or, perhaps, use another skip mechanism that would make this test no longer show up as "skipped" if we truly cannot enable it without it becoming flaky.

cc @DrewKimball in case you have some thoughts

Jira issue: CRDB-29525

[DrewKimball]

I never actually tried enabling it, just didn't want to add a flaky test. We could try enabling with stress and race disabled, but I worry it'd flake eventually. I guess we could just increase the allowed bounds until it no longer flakes (if it flakes).

[aadityasondhi]

I think the test flaked on my branch: https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelEssentialCi/10889866?hideProblemsFromDependencies=false&hideTestsFromDependencies=false&expandBuildChangesSection=true&expandBuildTestsSection=true&expandBuildProblemsSection=true

PR: #106722

[yuzefovich]

Hm, looks like the timeout occurred when the test was executing one of the test cases:

github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.RowsToStrMatrix(0xc00d645d70?)
	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:255 +0x107
github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).QueryStr(0x788d850?, {0x7871a80?, 0xc002060ea0}, {0x63c5df7, 0x57}, {0x0, 0x0, 0x0})
	github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:234 +0x11c
github.com/cockroachdb/cockroach/pkg/ccl/multitenantccl/tenantcostclient_test.TestEstimateQueryRUConsumption(0xc002060ea0)
	github.com/cockroachdb/cockroach/pkg/ccl/multitenantccl/tenantcostclient_test/pkg/ccl/multitenantccl/tenantcostclient/query_ru_estimate_test.go:179 +0x99b

and in the goroutine dump we see that the query is blocked waiting on network:

goroutine 5650 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc00bd4e360)
	google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/transport.go:328 +0x7c
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
	google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/transport.go:343
google.golang.org/grpc.(*csAttempt).recvMsg(0xc00733be10, {0x5fc5680?, 0xc000c0bf80}, 0xc00927fe60?)
	google.golang.org/grpc/external/org_golang_google_grpc/stream.go:1046 +0xc5
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x0?)
	google.golang.org/grpc/external/org_golang_google_grpc/stream.go:900 +0x25
google.golang.org/grpc.(*clientStream).withRetry(0xc00bd4e120, 0xc0066990b8, 0xc006699088)
	google.golang.org/grpc/external/org_golang_google_grpc/stream.go:751 +0x144
google.golang.org/grpc.(*clientStream).RecvMsg(0xc00bd4e120, {0x5fc5680?, 0xc000c0bf80?})
	google.golang.org/grpc/external/org_golang_google_grpc/stream.go:899 +0x12e
google.golang.org/grpc.invoke({0x78b79c0?, 0xc00b7eed50?}, {0x625822f?, 0x2?}, {0x61285c0, 0xc00927fe60}, {0x5fc5680, 0xc000c0bf80}, 0x0?, {0xc004be0b80, ...})
	google.golang.org/grpc/external/org_golang_google_grpc/call.go:73 +0xd7
github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2({0x78b79c0, 0xc00b7eed50}, {0x625822f, 0x21}, {0x61285c0, 0xc00927fe60}, {0x5fc5680, 0xc000c0bf80}, 0x7e235b249769edad?, 0x65b8aa0, ...)
	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:248 +0x424
google.golang.org/grpc.(*ClientConn).Invoke(0x496219688e106cad?, {0x78b79c0?, 0xc00b7eed50?}, {0x625822f?, 0x0?}, {0x61285c0?, 0xc00927fe60?}, {0x5fc5680?, 0xc000c0bf80?}, {0x0, ...})
	google.golang.org/grpc/external/org_golang_google_grpc/call.go:35 +0x223
github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*internalClient).Batch(0xc0084fa650, {0x78b79c0, 0xc00b7eed50}, 0x0?, {0x0, 0x0, 0x0})
...
github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc006e0a700?, {0x78b79c0?, 0xc001201d70?}, {0xc002868009?}, 0x4?, 0xc001201c20?)
	github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:953 +0xe6
github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1()
	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:260 +0x446
created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync
	github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:171 +0x22a

I don't quite understand why this would occur, perhaps on the overloaded CI machine the test takes long time? I might just increase the timeout since I don't see any signs of deadlock or something like that.

[yuzefovich]

There are some interesting logs related to the tenant though:

I230712 21:52:52.654612 5650 5@util/log/event_log.go:32 ⋮ [T10,nsql1,client=127.0.0.1:53778,hostssl,user=root] 256 ={"Timestamp":1689198772635843137,"EventType":"create_table","Statement":"CREATE TABLE ‹defaultdb›.public.‹abcd› (‹a› INT8, ‹b› INT8, ‹c› INT8, ‹d› INT8, INDEX (‹a›, ‹b›, ‹c›))","Tag":"CREATE TABLE","User":"root","DescriptorID":104,"TableName":"‹defaultdb.public.abcd›"}
I230712 21:52:52.822313 582 kv/kvserver/queue.go:616 ⋮ [T1,n1,s1,r52/1:‹/Table/5{1-2}›,raft] 257  rate limited in MaybeAdd (merge): throttled on async limiting semaphore
W230712 21:52:53.612281 5650 kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:637 ⋮ [T10,nsql1,client=127.0.0.1:53778,hostssl,user=root] 258  a transaction has hit the intent tracking limit (kv.transaction.max_intents_bytes); is it a bulk operation? Intent cleanup will be slower. txn: "sql txn" meta={id=e7519a1f key=/Tenant/10/Table/104/1/‹881879851610341377›/‹0› iso=Serializable pri=0.04449817 epo=0 ts=1689198772.655347726,0 min=1689198772.655347726,0 seq=46520} lock=true stat=PENDING rts=1689198772.655347726,0 wto=false gul=1689198773.155347726,0 ba: ‹5815 CPut, 5815 InitPut›
...
W230712 21:55:18.877732 5650 kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:637 ⋮ [T10,nsql1,client=127.0.0.1:53778,hostssl,user=root] 293  a transaction has hit the intent tracking limit (kv.transaction.max_intents_bytes); is it a bulk operation? Intent cleanup will be slower. txn: "sql txn" meta={id=1b1d7155 key=/Tenant/10/Table/104/1/‹881880325101846529›/‹0› iso=Serializable pri=0.01728139 epo=0 ts=1689198917.181285140,0 min=1689198917.181285140,0 seq=46520} lock=true stat=PENDING rts=1689198917.181285140,0 wto=false gul=1689198917.681285140,0 ba: ‹5815 CPut, 5815 InitPut›

Let's bump the cluster setting while also reducing the number of rows we're inserting in this test.

[yuzefovich]

Nevermind, it's the randomization of default-batch-bytes-limit that makes the queries much slower. The test needs to force some production values.

[msbutler]

This test just flaked on me on master.