Etag config TLS

[rfairburn]

This PR adds ETag/HTTP 304 support to the config TLS feature of osquery with an opt out config option. In deployments where you want fast convergence of configuration changes (say every minute), configs can become a HUGE percentage of all bandwidth uses. This can lower that by over 99% if you don't often change configs but just want them picked up immediately.

Since we only submit an "If-None-Match" if we receive an ETag, it shouldn't break any existing implementations, but I added an opt-out just-in case.

The opt-out can be removed if it doesn't make sense.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: rfairburn / name: Robert Fairburn (5723a8c, 74867a6)

[Copilot]

Pull request overview

Adds ETag / HTTP 304 handling to the TLS config plugin so repeated config refreshes can avoid downloading unchanged configs, with a --config_tls_etag opt-out. This extends the remote request stack to surface HTTP status codes and response headers, and updates the test TLS HTTP server plus unit tests to exercise the new behavior.

Changes:

• Add ETag caching in TLSConfigPlugin (send If-None-Match, reuse cached config on 304) guarded by --config_tls_etag.
• Extend TLSRequestHelper, Request, and TLSTransport to expose response status code and headers (ETag) to callers.
• Add/expand test server controls and new unit tests covering 200→304, payload changes, disabled mode, node API behavior, and error handling.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tools/tests/test_http_server.py	Adds ETag/304 support and test control endpoints to the TLS test server.
plugins/config/tls_config.h	Adds cached ETag/config fields and a mutex to serialize genConfig().
plugins/config/tls_config.cpp	Implements ETag-based conditional requests and 304 cache replay for TLS config fetches.
plugins/config/tests/tls_config_tests.cpp	Adds unit tests validating ETag and 304 behavior for TLS config.
osquery/remote/utility.h	Extends TLSRequestHelper API to optionally send ETag and return response metadata.
osquery/remote/transports/tls.h	Exposes response status code and header accessors on TLSTransport.
osquery/remote/transports/tls.cpp	Adds If-None-Match header injection when an ETag option is set.
osquery/remote/requests.h	Plumbs response status/header access through the Request wrapper.
osquery/remote/http_client.h	Adds header() accessor and makes status() const on HTTP_Response.
osquery/remote/http_client.cpp	Stops treating HTTP 304 as a redirect-like status in the redirect switch.
osquery/config/tests/config_tests.cpp	Adds a regression test ensuring rejected config updates do not alter hashes.
osquery/config/config.cpp	Moves hash computation to after validation and adjusts parsing guard.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[lucasmrod]

Worth checking this.

[rfairburn]

Reworked this PR based on design feedback: the etag now travels in the JSON request/response bodies instead of HTTP ETag/If-None-Match headers, since the config request is a POST where conditional-request semantics don't cleanly apply — all changes to the remote/request layer are gone, and {"etag":"ok"} (HTTP 200) replaces the 304 handling. The plugin also no longer replays a cached config on a match: Config::refresh() now understands an "unchanged" result and skips the reload entirely, and a new configApplied hook ensures a config that fails to apply is never acknowledged, so the failure stays visible without re-downloading an unchanged payload.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (2)

plugins/config/tls_config.cpp:133

• The error message for a server returning the reserved "ok" etag is misleading: the client does send an "etag" field (it’s just empty when there is no prior validator). This makes debugging server/client behavior harder and contradicts the conditional-request contract described in docs.

        if (etag == kEtagNotModified) {
          if (etag_.empty()) {
            return Status::failure(
                "Server signaled an unchanged config to a request that sent "
                "no etag");

osquery/config/config.cpp:767

• updateSource now treats an empty config string as a valid empty JSON object because parsing is skipped when clone is empty. This means an empty/partial response could wipe the active config instead of failing the refresh (previous behavior). Also, since comments are stripped into clone, hashing json (the unstripped content) reintroduces unnecessary reloads for comment-only changes.

  if ((!clone.empty() && !doc.fromString(clone, JSON::ParseMode::Iterative)) ||
      !doc.doc().IsObject()) {
    return Status::failure("Error parsing the config JSON");

[lucasmrod]

LGTM! Left some comments/questions.

[lucasmrod]

Worth checking this.

[rfairburn]

@lucasmrod I believe that I have backed out the code move and stored it in a separate branch. I have a separate bug report I can file for it separately.

The http rest finding by copilot was also addressed. If all automated tests pass here, I think this addresses everything you brought up.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (3)

osquery/config/config.cpp:553

• Config::refresh treats Status code 2 as "unchanged" success unconditionally, even during the very first refresh (Config::load calls refresh()). If a misbehaving/older external config plugin returns code 2 before any config has ever been applied, osquery will mark the config as loaded/valid while skipping update, leaving the process running without a real configuration.

  if (status.getCode() == 2) {
    // The plugin reports that the configuration is unchanged since it was
    // last applied; there is nothing to reload. Plugins may only report
    // this after a configuration was applied in this process lifetime.
    if (getRefresh() != FLAGS_config_refresh) {

plugins/config/tls_config.cpp:134

• The failure message says the request "sent no etag", but in this code path the client explicitly sent the etag field (it was just empty). Wording this as "empty etag" avoids confusion when debugging server/client behavior.

            return Status::failure(
                "Server signaled an unchanged config to a request that sent "
                "no etag");
          }

docs/wiki/deployment/remote.md:99

• PR description mentions "HTTP 304" support, but this implementation/documentation makes conditional config semantics part of the JSON body and does not use HTTP 304 / If-None-Match headers. Calling out 304 explicitly here would prevent confusion for server implementers reading the docs.

Configurations rarely change between refreshes, so the **tls** config plugin supports conditional requests carried in the JSON bodies (the request is a `POST`, so HTTP `ETag`/`If-None-Match` header semantics do not apply). Servers are free to ignore all of this: a response without an `"etag"` key is treated as the configuration content, byte for byte, exactly as before.