kube-apiserver oom, list resource consume too much memory cause json decode

[V0idk]

What happened?

The APIServer concurrency capability is too weak. In the test, the memory usage of 20 concurrent requests increases to 12 GB. The data size of "kubectl get crd -A -o yaml" is 20 MB.

1. Why does serialization consume so much memory? Is there any optimization mechanism?
2. Another point to note: when I call 100 watches concurrently（and we know that watch will initially treat all items as add events, which are equivalent to lists.）, kube-apiserver only goes up to 3GB, while 10 LIST concurrent operations go up to 8GB. Why is the memory usage difference between watches and lists so huge?

after i specfic resourceVersion=0, memory usage does not improve:

What did you expect to happen?

Why does serialization consume so much memory? Is there any optimization mechanism?

How can we reproduce it (as minimally and precisely as possible)?

just kubectl get crd -A -o yaml & concurrently

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
# paste output here

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[V0idk]

/sig api-machinery

[nayihz]

Can this issue be reproduced? What's your k8s cluster version? Do you try to reproduce it in other cluster?

[mauri870]

Additionally, can you share the actual profile file? Thanks.

[seans3]

/triage accepted

/assign @benluddy

[benluddy]

/cc @p0lyn0mial

Each watch event is serialized to a buffer and flushed to the response one at a time, whereas for lists, complete list objects are serialized to a buffer before writing to the response. I don't think truly streaming encoders are coming soon, but https://kep.k8s.io/3157 is targeting beta in v1.31 and will allow "lists" to be performed using (and have approximately the same cost as) the watch mechanism.

[DrAuYueng]

/cc