Osquery perf split common software between hosts

[ksykulev]

If the total requested common software is 10,000 and there are 200 hosts. Instead of all hosts having 10,000 pieces of software, each host will have a 50 software slice of the 10,000 total requested. This is controlled by common_software_count. If you wish to have each host have a certain amount of software, use the unique_software_count

Related issue: Resolves #33668

[codecov]

Codecov Report

❌ Patch coverage is 0% with 87 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.18%. Comparing base (9e3cab6) to head (0a01584).
⚠️ Report is 17 commits behind head on main.

Files with missing lines	Patch %	Lines
cmd/osquery-perf/agent.go	0.00%	87 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #33958      +/-   ##
==========================================
- Coverage   64.18%   64.18%   -0.01%     
==========================================
  Files        2054     2055       +1     
  Lines      206397   206521     +124     
  Branches     6794     6794              
==========================================
+ Hits       132477   132550      +73     
- Misses      63518    63568      +50     
- Partials    10402    10403       +1     

Flag	Coverage Δ
backend	65.28% <0.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[sharon-fdm]

I will approve assuming the answer to Lucas' comment is positive.

@ksykulev , if not, please fix and Slack for me to reapprove.

[lucasmrod]

To check the math, could you explain what are the expected values for this load test for these flags?

• --common_software_count
• --duplicate_bundle_identifiers_percent

[ksykulev]

Here are the params I want to use.

  --host_count 500
  --total_host_count 50000
  --host_index_offset [varies per container: 0, 500, 1000, ..., 49500]
  --common_software_count 50000
  --duplicate_bundle_identifiers_percent 1000

This will generate 500 hosts per container. I want a total of 50000 pieces of common software and 1000% duplicate. So:

• Common software: 50,000
• Duplicate software: 50,000 × 1000 / 100 = 500,000
• Total: 550,000 software entries to distribute
• perHostCount = 550,000 / 50,000 hosts = 11 software entries per host
• Each bundle ID appears 11 times on average (1 Common + 10 Duplicates). So when the rename happens there will be a total of 10 renames happening from hosts who have duplicate bundle ids (hosts 5,000 - 50,000 will be doing the renaming). This is where the concurrency load test comes in, because all of these (approximately) 45k hosts will be trying to renaming fairly close to the same time.

Container 0 (host_index_offset = 0)

Agent 1
- globalAgentIndex = 0 + (1-1) = 0
- startIdx = 0 × 11 = 0
- endIdx = 0 + 11 = 11
- Gets software indices 0-10
...
Agent 500
- Agent 500:
- globalAgentIndex = 0 + (500-1) = 499
- startIdx = 499 × 11 = 5,489
- endIdx = 5,489 + 11 = 5,500
- Gets software indices 5,489-5,499

Container 10 (host_index_offset = 5000):

Agent 1:
- globalAgentIndex = 5000 + (1-1) = 5,000
- startIdx = 5,000 × 11 = 55,000
- endIdx = 55,000 + 11 = 55,011
- Gets software indices 55,000-55,010

Container 99 (host_index_offset = 49500):

Agent 1:
    - globalAgentIndex = 49500 + (1-1) = 49,500
    - startIdx = 49,500 × 11 = 544,500
    - endIdx = 544,500 + 11 = 544,511
    - Gets software indices 544,500-544,510

[lucasmrod]

Agents with software indices >= 50_000 will trigger the heavy renaming, right?
Sorry, I'm trying to understand the intent.

Overall looks good.

[ksykulev]

Exactly! So round one we run osquery perf. It generates a bunch of common software and software with duplicate bundle ids.
"com.fleetdm.osquery-perf.common_0"
"com.fleetdm.osquery-perf.common_1"
...
"com.fleetdm.osquery-perf.common_N"

We need to run osquery perf again with the softwareRenaming flag. This will take all the software with duplicate bundle ids and force a rename. This is precisely the reason we need a stable bundle id generation, and the reason for the need for a globalAgentIndex. Between osquery perf run we need to generate exactly the same bundle ids. (otherwise we could just use a hashing function to generate unique bundle ids and avoid all this complexity.)

[lucasmrod]

LGTM. Left one final question.