sql/colexecerror: CatchVectorizedRuntimeError uses an expensive debug.Stack call

[rafiss]

Describe the problem

The usage of debug.Stack here is problematic:

cockroach/pkg/sql/colexecerror/error.go

Line 41 in 81569be

stackTrace := string(debug.Stack())

During a recent customer outage, we captured this profile:

(Full profile available here.)

In the outage, the CRDB behavior was that once CPU usage became high enough, response times started to go up, and execution throughput went down. Notably, the throughput/latency impact is correlated with a large increase in concurrent open transactions. See the support ticket for more details.

The main part of the issue here appears to be that the usage of debug.Stack causes lock contention due to the system calls that rely on futex.

To Reproduce

This can be seen with the admission-control/tpcc-severe-overload test. See these notes.

Expected behavior

Make the CatchVectorizedRuntimeError less expensive.

There are two main ideas:

• Instead of inspecting the stack trace in the common case, we should be able to first check for specific error types. Note that ExpectedError already wraps many errors with a known type:

  cockroach/pkg/sql/colexecerror/error.go

  Lines 202 to 204 in 81569be

  	func ExpectedError(err error) {
  	panic(newNotInternalError(err))
  	}

  . The related InternalError function could do the same.
• In the fallback case where we must inspect the stack, use runtime.Callers and runtime.CallersFrames instead of debug.Stack. It should be much cheaper. (See runtime: Stack() calls lock against each other unnecessarily, even when single-goroutine golang/go#56400 for some discussion on why debug.Stack is slower.)

In addition to these two changes, there might be ways to improve the function even more. We can use this issue to track those ideas as well.

@michae2 has some early benchmark results from making some of these changes.

Jira issue: CRDB-38252