Reduced redundant database calls in the osquery distributed query results hot path

[getvictor]

Related issue: Resolves #42156

The core change: instead of loading AppConfig, HostFeatures, TeamMDMConfig, and rebuilding the detail query map independently inside each call to directIngestDetailQuery and ingestDetailQuery (so ~2N times per check-in with N detail results), we load everything once into a hostDetailQueryConfig struct and pass it through.

Before

SubmitDistributedQueryResults loop:
  for each query result:
    → ingestQueryResults
      → directIngestDetailQuery:  loads AppConfig, HostFeatures, TeamMDMConfig, builds detail query map
      → ingestDetailQuery:        loads AppConfig, HostFeatures, TeamMDMConfig, builds detail query map
  after loop:
    loads AppConfig for labels/policies
    loads AppConfig AGAIN for deferred host save

After

SubmitDistributedQueryResults loop:
  on first detail query result:
    → loadHostDetailQueryConfig: loads AppConfig, HostFeatures, TeamMDMConfig, builds detail query map ONCE
  for each query result:
    → ingestQueryResults (receives pre-loaded config)
      → directIngestDetailQuery: just looks up the query in the cached map
      → ingestDetailQuery:       just looks up the query in the cached map
  after loop:
    loads AppConfig once for labels/policies/deferred host save

The detail config is lazy-loaded — if a check-in only has label/policy results and no detail queries, the HostFeatures/TeamMDMConfig calls are skipped entirely.

Other changes bundled in

1. serialUpdateHost now receives the request context and uses context.WithoutCancel(ctx) instead of context.Background(), so the background goroutine preserves OTEL traces and logging context without being subject to request cancellation.

2. Deferred save host at the end of SubmitDistributedQueryResults reuses the already-loaded AppConfig instead of loading it a third time. The old code silently skipped the host save if that third load failed.

Checklist for submitter

If some of the following don't apply, delete the relevant line.

• Changes file added for user-visible changes in changes/, orbit/changes/ or ee/fleetd-chrome/changes.

Testing

• QA'd all new/changed functionality manually
  • Ran a local load test with osquery perf

Summary by CodeRabbit

• Performance
  • Optimized distributed query result processing by preloading configuration once per request instead of repeatedly per query result, reducing redundant database calls and improving overall query performance.

[getvictor]

@coderabbitai full review

[coderabbitai]

✅ Actions performed

Full review triggered.

[Copilot]

Pull request overview

This PR refactors the osquery distributed results ingestion path to reduce redundant configuration/datastore lookups during detail query processing, improving scalability on a very hot server code path.

Changes:

• Introduces a hostDetailQueryConfig struct and loadHostDetailQueryConfig helper to pre-load configuration for detail query building/ingestion.
• Plumbs the pre-loaded config through SubmitDistributedQueryResults → ingestQueryResults → detail ingestion helpers to avoid per-result reloading.
• Adds a changelog entry describing the reduced redundant DB calls.

Reviewed changes

Copilot reviewed 1 out of 2 changed files in this pull request and generated 3 comments.

File	Description
server/service/osquery.go	Preloads and threads detail-ingestion configuration through the distributed results ingestion path to reduce redundant datastore calls.
changes/35467-detail-query-config-preload	Documents the performance improvement in the changelog.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[coderabbitai]

Walkthrough

The pull request introduces configuration pre-loading to the osquery distributed query detail query ingestion hot path. A new hostDetailQueryConfig struct and loadHostDetailQueryConfig() helper are added to load AppConfig, HostFeatures, TeamMDMConfig, and conditional access configuration once per request. This preloaded configuration is then passed through the detail query ingestion pipeline, replacing individual database fetches at each call site. The change consolidates configuration loading across SubmitDistributedQueryResults, ingestQueryResults, and related detail ingestion functions.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: reducing redundant database calls in the osquery distributed query results hot path, which aligns with the core objective.
Linked Issues check	✅ Passed	The code changes implement all core engineering objectives from #42156: preloading hostDetailQueryConfig once per request, reducing AppConfig/HostFeatures/TeamMDMConfig/conditional access lookups from ~2N to 1, and eliminating GetDetailQueries rebuilds in ingestion.
Out of Scope Changes check	✅ Passed	All changes are scoped to the osquery detail query ingestion hot path (server/service/osquery.go and changes file). No unrelated modifications to public APIs, schemas, or unrelated subsystems detected.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description check	✅ Passed	PR description is comprehensive with related issue, clear before/after code comparisons, bundled changes explained, and testing verification provided.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch victor/42156-detail-query

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Tip

CodeRabbit can generate a title for your PR based on the changes with custom instructions.

Set the reviews.auto_title_instructions setting to generate a title for your PR based on the changes in the PR with custom instructions.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@server/service/osquery.go`:
- Around line 723-728: GetDetailQueries is still being reconstructed on the hot
path because hostDetailQueryConfig currently only caches inputs (appConfig,
features, mdmTeamConfig, conditionalAccessMicrosoftEnabled) but not the resolved
detail-query map; functions detailQueriesForHost, directIngestDetailQuery, and
ingestDetailQuery rebuild the full map repeatedly. Fix by adding a cached
resolved map field (e.g., detailQueries map[string]DetailQuery or similar) and a
sync.Once or lazy getter method on hostDetailQueryConfig (e.g.,
getDetailQueries()) that builds and stores the map once from GetDetailQueries
and returns the cached map thereafter; update detailQueriesForHost,
directIngestDetailQuery, and ingestDetailQuery to call that getter instead of
calling GetDetailQueries directly so the expensive build happens once per config
instance.
- Around line 1113-1120: The current call to loadHostDetailQueryConfig (stored
in detailConfig / ac) runs on every distributed/write request; move that call so
it is lazy-loaded only when handling a detail result whose name begins with
"fleet_detail_query_" (i.e., inside the branch that processes detail query
results), rather than before processing all results. Keep AppConfig loading
separate for the label/policy/live-query save paths so those flows do not
trigger HostFeatures/TeamMDMConfig work; when you relocate the call, ensure any
error returned by loadHostDetailQueryConfig only aborts the detail ingestion
path and not the overall submit flow. Locate references to detailConfig, ac, and
loadHostDetailQueryConfig in the distributed/write handling code and adjust
control flow and error handling accordingly.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 80993c46-4012-4b85-b914-571ff3cc4045

📥 Commits

Reviewing files that changed from the base of the PR and between 7eb07b4 and 81e25d0.

📒 Files selected for processing (2)

• changes/35467-detail-query-config-preload
• server/service/osquery.go

[codecov]

Codecov Report

❌ Patch coverage is 63.63636% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.48%. Comparing base (7eb07b4) to head (9e56c74).
⚠️ Report is 12 commits behind head on main.

Files with missing lines	Patch %	Lines
server/service/osquery.go	63.63%	13 Missing and 7 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #42157      +/-   ##
==========================================
- Coverage   66.49%   66.48%   -0.01%     
==========================================
  Files        2516     2516              
  Lines      201987   202026      +39     
  Branches     9071     9071              
==========================================
+ Hits       134309   134316       +7     
- Misses      55545    55576      +31     
- Partials    12133    12134       +1     

Flag	Coverage Δ
backend	68.27% <63.63%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review.

[getvictor]

Hi @lucasmrod, I'm assigning this PR review to you since this is in orchestration domain. Thank you.