Optimized api/latest/fleet/software/titles endpoint#40458
Conversation
|
@coderabbitai full review |
✅ Actions performedFull review triggered. |
There was a problem hiding this comment.
Pull request overview
This PR optimizes the /api/latest/fleet/software/titles endpoint to address severe performance issues (queries taking 40+ seconds at scale with 300K software titles and 95K hosts). The optimization introduces a two-phase query strategy for the common case of listing titles ordered by hosts_count without additional filters.
Changes:
- Introduces an optimized query path that uses a covering index scan followed by enrichment of only the paginated results
- Parallelizes list and count queries using errgroup to further improve performance
- Refactors shared post-processing logic into a dedicated function
- Updates the database index on
software_titles_host_countsto includeglobal_statsfor better query optimization
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| server/datastore/mysql/software_titles.go | Implements two-phase optimized query path with covering index scan, parallel list/count queries using errgroup, and refactored post-processing logic |
| server/datastore/mysql/migrations/tables/20260225000000_OptimizeSoftwareTitlesHostCountsIndex.go | Database migration to replace index with optimized version including global_stats column |
| server/datastore/mysql/schema.sql | Schema updates reflecting new index and migration counter |
| changes/35799-software-titles | User-facing changelog entry |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
WalkthroughThis PR optimizes the Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@server/datastore/mysql/software_titles.go`:
- Around line 273-289: The count query is using mutated pagination arguments
because appendListOptionsWithCursorToSQL modifies getTitlesStmt and appends to
args; fix by creating the count SQL and its own args before mutating args—for
example, capture getTitlesCountStmt and a copy of args (e.g., countArgs :=
append([]interface{}{}, args...)) prior to calling
appendListOptionsWithCursorToSQL and spliceSecondaryOrderBySoftwareTitlesSQL,
then use countArgs with sqlx.GetContext for the count goroutine while using the
mutated getTitlesStmt and args for sqlx.SelectContext; update references to
getTitlesStmt, getTitlesCountStmt, args, and the two goroutines to use the
separate arg slices.
ℹ️ Review info
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
changes/35799-software-titlesserver/datastore/mysql/migrations/tables/20260225000000_OptimizeSoftwareTitlesHostCountsIndex.goserver/datastore/mysql/schema.sqlserver/datastore/mysql/software_titles.go
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #40458 +/- ##
==========================================
- Coverage 66.29% 66.24% -0.05%
==========================================
Files 2467 2467
Lines 197538 197561 +23
Branches 8618 8585 -33
==========================================
- Hits 130952 130875 -77
- Misses 54736 54808 +72
- Partials 11850 11878 +28
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
# Conflicts: # server/datastore/mysql/schema.sql
Related issue: Resolves #35799
Loadtest results for 100K hosts and 300K software titles.
Documented in the issue:
The fix includes a small behavior change. The default primary sort of /software/titles remains host_counts, but the secondary sort is now software_title_id and not name. This was necessary to optimize the endpoint. This means that if you have 1 host in your fleet, the software page will not show the software titles ordered by name anymore. For large fleets, this does not matter since all titles generally have different host counts.
Checklist for submitter
changes/,orbit/changes/oree/fleetd-chrome/changes.Testing
Database migrations
Summary by CodeRabbit