K8s trace context for APIServer is incorrect

[CasperLiu]

What happened?

In Kubernetes release 1.28, the tracing structure for the API server is depicted as follows:

Within the API server's code structure, the spans "List(recursive=true) etcd3" and "SerializeObject" are expected to be nested under the "List" span. However, "SerializeObject" appears parallel to "List", which is not the intended behavior.

staging/src/k8s.io/apiserver/pkg/endpoints/handlers/get.go

func ListResource(r rest.Lister, rw rest.Watcher, scope *RequestScope, forceWatch bool, minRequestTimeout time.Duration) http.HandlerFunc {
	return func(w http.ResponseWriter, req *http.Request) {
		ctx := req.Context()
		// For performance tracking purposes.

                 // CasperLiu: this span is for high level - "List"
		ctx, span := tracing.Start(ctx, "List", traceFields(req)...)

		namespace, err := scope.Namer.Namespace(req)
		if err != nil {
			scope.err(err, w, req)
			return
		}

                 // omit...
		span.AddEvent("About to List from storage")
                 // CasperLiu: span inside below function is for low level - "List(recursive=true) etcd3"
		result, err := r.List(ctx, &opts)
		if err != nil {
			scope.err(err, w, req)
			return
		}
		span.AddEvent("Listing from storage done")
		defer span.AddEvent("Writing http response done", attribute.Int("count", meta.LenList(result)))
                 // CasperLiu: span inside below function is for low level - "SerializeObject"
		transformResponseObject(ctx, scope, req, w, http.StatusOK, outputMediaType, result)
	}
}

What did you expect to happen?

Within the API server's code structure, the spans "List(recursive=true) etcd3" and "SerializeObject" are expected to be nested under the "List" span.

How can we reproduce it (as minimally and precisely as possible)?

1. Configure the Kubernetes API server to enable tracing and direct the emitted spans to a backend.
2. Illustrate the resulting trace flow.
3. Invoke a quorum read LIST request for Kubernetes resources; within the trace flow, you may identify instances of incorrect trace context.

Anything else we need to know?

No response

Kubernetes version

kuberenetes version:
1.28.3

Cloud provider

Alibaba Cloud Kubernetes (ACK)

OS version

Unrelated to the OS version

Install tools

No need

Container runtime (CRI) and version (if applicable)

Unrelated to the CRI version

Related plugins (CNI, CSI, ...) and versions (if applicable)

Unrelated to the plugins version

[pranav-pandey0804]

/sig instrumentation

[CasperLiu]

I have made a code change to resolve this issue, PR is as blow:
#124079

[dgrisonnet]

/assign @CasperLiu
/triage accepted