Skip to content

Fix label membership being cleared when a label query errors - #49403

Merged
nulmete merged 2 commits into
mainfrom
46399-query-errors-can-cause-label
Jul 20, 2026
Merged

Fix label membership being cleared when a label query errors#49403
nulmete merged 2 commits into
mainfrom
46399-query-errors-can-cause-label

Conversation

@nulmete

@nulmete nulmete commented Jul 16, 2026

Copy link
Copy Markdown
Member

Related issue: Resolves #46399

When a label's query errors on a host (e.g. the extension socket is unavailable) instead of returning zero rows, Fleet was recording that error the same as a definitive "no match," clearing the host's existing label membership. This could unintentionally remove configuration profiles or other automations scoped to that label. The fix leaves existing label membership untouched when a label query errors.

Checklist for submitter

  • Changes file added for user-visible changes in changes/, orbit/changes/ or ee/fleetd-chrome/changes.
    See Changes files for more information.

Testing

  • Added/updated automated tests

  • QA'd all new/changed functionality manually

Setup: macOS VM enrolled as a Fleet host, with a dynamic label whose query targets a real, always-present table but with a deliberately invalid WHERE clause, so the query fails deterministically (a no such column SQL error).

-- working version (label matches)
SELECT * FROM os_version;

-- broken version (query errors on every run)
SELECT * FROM os_version WHERE this_column_does_not_exist = 1;

Before (bug reproduced on unpatched code)

  1. Set the label's query to the working version and refetched the host — confirmed it shows up under the host's Labels.
  2. Edited the label's query to the broken version.
  3. Clicked Refetch on the host.
  4. Result: the label disappeared from the host's Labels list — a query error incorrectly cleared existing membership.

After (fix verified)

  1. Reset the label's query to the working version and refetched — confirmed membership was restored.
  2. Edited the label's query to the broken version again.
  3. Clicked Refetch on the host.
  4. Result: the label remained on the host's Labels list — a query error now correctly leaves existing membership untouched.

Summary by CodeRabbit

  • Bug Fixes
    • Preserved existing dynamic label memberships when label queries fail or yield unknown results.
    • Avoided treating unknown/failed evaluations as label removals.
    • Ensured label updates/removals are applied only when a definite match or non-match is returned.
  • Tests
    • Expanded coverage for label query errors across datastore, async processing, and distributed execution to confirm memberships remain unchanged.
    • Updated expectations for queued async updates to skip errored labels.

A label query that errors (e.g. extension socket unavailable) was being
recorded the same as a query that ran fine and returned zero rows,
causing existing label membership to be incorrectly removed. Query
errors now leave existing membership untouched.
@nulmete

nulmete commented Jul 16, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.

@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

Label query errors are now represented as unknown results rather than non-matches. Asynchronous recording skips unknown results and avoids empty Redis updates. MySQL recording also skips unknown matches, preserving existing label_membership rows. Tests cover distributed query errors, Redis queue contents, and membership retention after a failed label query.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The changes and tests satisfy #46399 by keeping existing label membership unchanged when a query result is nil/error.
Out of Scope Changes check ✅ Passed The modified datastore, async service, and tests all directly support the label-error membership fix.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Title check ✅ Passed The title clearly summarizes the main fix: preserving label membership when a label query errors.
Description check ✅ Passed The description includes the related issue, the bug context, the required checklist items, and clear testing/QA details.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch 46399-query-errors-can-cause-label

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@nulmete
nulmete marked this pull request as ready for review July 16, 2026 12:18
@nulmete
nulmete requested a review from a team as a code owner July 16, 2026 12:18
Copilot AI review requested due to automatic review settings July 16, 2026 12:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

  • Copilot's review of this pull request may be incomplete because some of the changed files are excluded by your Copilot content exclusion settings. See Excluding content from Copilot for details.

Pull request overview

This PR fixes dynamic label evaluation so that when a label query returns an error status (instead of a definitive “0 rows”), Fleet records the result as unknown and does not clear existing label membership—preventing unintended removal of label-scoped automation (resolves #46399).

Changes:

  • Treat errored label query executions as nil/unknown and skip membership deletes/updates for those labels (MySQL + async Redis path).
  • Update async Redis script/argument building to avoid writing “delete” entries for errored label results.
  • Add/extend tests to validate that query errors preserve existing label membership.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
server/service/osquery_test.go Adds coverage ensuring query error status is recorded as nil label result (unknown).
server/service/async/async_label.go Skips nil label results when writing async label membership ops to Redis; guards ZADD when args are empty.
server/service/async/async_label_test.go Updates expectations so nil results are skipped (not treated as deletes).
server/datastore/mysql/labels.go Skips updates/deletes when label result is nil (unknown/error), preserving membership.
server/datastore/mysql/labels_test.go Adds a regression test ensuring nil results keep existing membership (but currently has compile errors: invalid new("1") / new(true)).
changes/46399-query-errors-can-cause-label-membership-unassigned.md Release note (content excluded by policy; not reviewed).
Files excluded by content exclusion policy (1)
  • changes/46399-query-errors-can-cause-label-membership-unassigned.md

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread server/datastore/mysql/labels_test.go
Comment thread server/datastore/mysql/labels_test.go
@codecov

codecov Bot commented Jul 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 67.80%. Comparing base (3e695c7) to head (86a9ea4).
⚠️ Report is 83 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #49403      +/-   ##
==========================================
- Coverage   68.15%   67.80%   -0.35%     
==========================================
  Files        3849     3881      +32     
  Lines      243458   247434    +3976     
  Branches    13110    13110              
==========================================
+ Hits       165934   167782    +1848     
- Misses      62526    64487    +1961     
- Partials    14998    15165     +167     
Flag Coverage Δ
backend 69.22% <100.00%> (-0.45%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

cdcme
cdcme previously approved these changes Jul 16, 2026
@lucasmrod lucasmrod self-assigned this Jul 17, 2026
lucasmrod
lucasmrod previously approved these changes Jul 17, 2026
The test used a nil result as shorthand for "remove label membership,"
relying on the exact behavior fixed in the previous commit. Use an
explicit false (non-match) instead, since nil now means "query
errored, leave membership unchanged."
@nulmete
nulmete dismissed stale reviews from lucasmrod and cdcme via 86a9ea4 July 20, 2026 13:13
@nulmete
nulmete merged commit 900c54e into main Jul 20, 2026
40 checks passed
@nulmete
nulmete deleted the 46399-query-errors-can-cause-label branch July 20, 2026 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Query errors can cause label membership to be unassigned

5 participants