Support gzip compression for API responses

[zwass]

Goal

User story
As a person who deploys Fleet,
I want to enable gzip compression on agent API responses
so that I can reduce the number of outbound bytes sent from the Fleet server (one of the biggest AWS expenses).

Changes

Product

This should be designed in a way that it can be progressively enabled until such time as we can enable it by default.

• Fleet's agent (fleetd) changes: Confirm/add support for gzip responses in fleetd
• Fleet server changes: Add support for gzip responses in Fleet API server
• Other reference documentation changes: Update Fleet server configuration docs if flags are introduced: Document server_gzip_responses configuration #38520
• UI changes: None
• CLI (fleetctl) usage changes: None
• YAML changes: None
• REST API changes: None
• GitOps mode UI changes: None
• GitOps generation changes: None
• Activity changes: None
• Permissions changes: None
• Changes to paid features or tiers: None
• My device and fleetdm.com/better changes: None
• Usage statistics: None
• First draft of test plan added
• Once shipped, requester has been notified
• Once shipped, dogfooding issue has been filed
  • @noahtalerman: Route this issue to :help-customers. In addition include these steps:
  • Turn this on for rocher
  • Incrementally turn this on for customers
  • Change the configuration option to default on
• Once shipped

Engineering

• Test plan is finalized
• Contributor API changes: None
• Feature guide changes: None
• Database schema migrations: None
• Load testing: Testing should be performed.
• Load testing/osquery-perf improvements: Ensure that osquery-perf can send the appropriate headers to enable gzip compression
• This is a premium only feature: No

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

QA

Risk assessment

• Requires testing in a hosted environment: Maybe. Could be useful to measure reduction in egress costs.
• Requires load testing: Yes
• Risk level: High
• Risk description: If this messes up agent communications it could cause outages up to P0

Test plan

Make sure to go through the list and consider all events that might be related to this story, so we catch edge cases earlier.

1. Verify in browser devtools that API requests (over 1kb responses, eg. hosts API) are gzipped when flag is enabled and not when flag is disabled.
2. Verify osqueryd communication still functions when flag is enabled. Use osqueryd >= 5.21.0 and the agent flag turned on (default in Orbit >= 1.52.0). Run a live query.
3. Verify Orbit communication still functions when flag is enabled. Run a script.

Testing notes

Agent load testing: Measure the CPU utilization of the agent with this capability off. Measure the utilization with this capability on. Is there a noticeable difference?

Confirmation

1. Engineer: Added comment to user story confirming successful completion of test plan.
2. QA: Added comment to user story confirming successful completion of test plan.

[zwass]

I propose that we implement this in a few phases

1. Enable the tls_accept_gzip flag by default with fleetd/osquery versions that support it. This should have little effect as the osqueryd agents will send the accept header but the server will ignore it and serve the same responses as usual.
2. Add a configuration option to the Fleet server to enable sending gzip responses (default off).
3. Turn this on in dogfood
4. Turn this on for rocher
5. Incrementally turn this on for customers.
6. Change the configuration option to default on (or remove it entirely?)