feat: accept protobuf responses where possible from kube-api (#18603)

[cnmcavoy]

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Toolchain Guide
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.
• Optional. My organization is added to USERS.md.
• Optional. For bug fixes, I've indicated what older releases this fix should be cherry-picked into (this may or may not happen depending on risk/complexity).

Fixes: #18603

[codecov]

Codecov Report

Attention: Patch coverage is 17.33333% with 62 lines in your changes missing coverage. Please review.

Project coverage is 50.34%. Comparing base (1aa898c) to head (5a30eb7).
Report is 5 commits behind head on master.

Files	Patch %	Lines
cmd/argocd-server/commands/argocd_server.go	0.00%	37 Missing ⚠️
pkg/apis/application/v1alpha1/types.go	0.00%	6 Missing and 1 partial ⚠️
cmd/argocd-dex/commands/argocd_dex.go	0.00%	6 Missing ⚠️
...ntroller/commands/argocd_application_controller.go	0.00%	4 Missing ⚠️
cmd/argocd-notification/commands/controller.go	0.00%	4 Missing ⚠️
...t-controller/commands/applicationset_controller.go	66.66%	1 Missing and 1 partial ⚠️
controller/appcontroller.go	80.00%	1 Missing ⚠️
controller/cache/cache.go	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #18602      +/-   ##
==========================================
- Coverage   50.36%   50.34%   -0.02%     
==========================================
  Files         315      315              
  Lines       43190    43219      +29     
==========================================
+ Hits        21752    21758       +6     
- Misses      18954    18975      +21     
- Partials     2484     2486       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jannfis]

I wonder how big of an improvement this is, @cnmcavoy. Do you have any relevant numbers please?

[chaochn47]

This is the result I got when experimenting on a 15k pods cluster before giving the advise to tweak the content type to proto.

Expect the diff would grow as least linearly when the number of pods grows.

(24-06-13 1:08:36) <0> [~/workplace/EKSKubernetes/src/EKSDataPlaneKubernetes/staging/src/k8s.io/client-go/examples/out-of-cluster-client-configuration]
dev-dsk-chaochn-2c-a26acd76 % ./app --use-proto=true
I0613 01:08:51.106629  140283 main.go:80] took 3.154472032s, pods resourceVersion 700889256, 15025 pods

(24-06-13 1:08:51) <0> [~/workplace/EKSKubernetes/src/EKSDataPlaneKubernetes/staging/src/k8s.io/client-go/examples/out-of-cluster-client-configuration]
dev-dsk-chaochn-2c-a26acd76 % ./app --use-proto=false
I0613 01:09:07.317080  140898 main.go:80] took 8.381112862s, pods resourceVersion 700895115, 15025 pods

(24-06-13 1:09:07) <0> [~/workplace/EKSKubernetes/src/EKSDataPlaneKubernetes/staging/src/k8s.io/client-go/examples/out-of-cluster-client-configuration]
dev-dsk-chaochn-2c-a26acd76 % ./app --use-proto=true
I0613 01:19:23.438321  149026 main.go:80] took 2.846576869s, pods resourceVersion 701210537, 15025 pods

(24-06-13 1:19:23) <0> [~/workplace/EKSKubernetes/src/EKSDataPlaneKubernetes/staging/src/k8s.io/client-go/examples/out-of-cluster-client-configuration]
dev-dsk-chaochn-2c-a26acd76 % ./app --use-proto=false
I0613 01:19:33.774061  149092 main.go:80] took 8.52089375s, pods resourceVersion 701214264, 15025 pods

(24-06-13 1:19:33) <0> [~/workplace/EKSKubernetes/src/EKSDataPlaneKubernetes/staging/src/k8s.io/client-go/examples/out-of-cluster-client-configuration]
dev-dsk-chaochn-2c-a26acd76 % ./app --use-proto=true
I0613 01:30:38.632484  157860 main.go:80] took 2.890275373s, pods resourceVersion 701554289, 15025 pods

(24-06-13 1:30:38) <0> [~/workplace/EKSKubernetes/src/EKSDataPlaneKubernetes/staging/src/k8s.io/client-go/examples/out-of-cluster-client-configuration]
dev-dsk-chaochn-2c-a26acd76 % ./app --use-proto=false
I0613 01:30:48.910746  157947 main.go:80] took 8.345562795s, pods resourceVersion 701557468, 15025 pods

[crenshaw-dev]

Tested against an internal instance, and I'm not seeing a significant difference in resource usage (memory, cpu, network).

I think part of the problem is that the PR only sets content types on some of the clients, mostly the ones that are used to get resources the component immediately needs from the local k8s API (i.e. things like secrets and configmaps).

The heavy lifting of the application controller is done with separately-initialized k8s clients in the cluster cache code. I tried to set that to use protobuf too here, but am so far not seeing significant wins: b98df97

[crenshaw-dev]

Here's unchanged vs. this PR vs. this PR + cluster cache change:

I'm surprised not to see any difference after the cluster cache change, but maybe there's just not enough activity on the instance for it to matter.

[cnmcavoy]

I think part of the problem is that the PR only sets content types on some of the clients, mostly the ones that are used to get resources the component immediately needs from the local k8s API (i.e. things like secrets and configmaps).

I updated the PR to ensure that the protobuf headers get set in the other places and also tried to wire up the user agents correctly. The cluster cache and several controllers were defaulting to the stock user agent, which made it hard to trace who was making the k8s api calls.