Perf test idea#203
Closed
StealthyCoder wants to merge 28 commits into
Closed
Conversation
We need the ability to create a token for e2e integration tests Signed-off-by: Andy Doan <andy@foundries.io>
This allows gen-certs to create devices faster using all the cpu cores. It also allows us to use the dg-sat binary rather than having to build the code. Signed-off-by: Andy Doan <andy@foundries.io>
This just preps a /data directory with things we'll need Signed-off-by: Andy Doan <andy@foundries.io>
Not super interesting but it helped find some suboptimal things we were doing with db.go setup Signed-off-by: Andy Doan <andy@foundries.io>
This adds a way to run device listing while registering devices to test concurrent reads along with writes. 5000 devices / spawn rate 80 ---------------------------- DG API: * 231 requests/second * 0 errors * 630ms (99%tile 3600ms) REST API * 455 requests/second * 0 errors * 160ms (99%tile 3300ms) Signed-off-by: Andy Doan <andy@foundries.io>
Signed-off-by: Andy Doan <andy@foundries.io>
Signed-off-by: Andy Doan <andy@foundries.io>
Signed-off-by: Andy Doan <andy@foundries.io>
Signed-off-by: Andy Doan <andy@foundries.io>
…o cert generator Existing openssl-fork based cert generation takes minutes for thousands of devices; a single in-process Go generator does 5000 ECDSA certs in ~250ms. Verified end-to-end (10 and 5000 devices): golang:1.24 builder bumped to 1.25 to match go.mod, setup.sh now runs auth-init before tuf-init (TUF needs the HMAC secret auth-init creates), and the locust service's /data mount is read-write so headless runs can write their CSV/HTML report.
contrib/perf-test is now the verified, self-contained replacement (faster Go-based cert generation, single docker-compose flow); keeping both around would leave two parallel, easily-diverging perf-testing setups.
These were how-to notes for the now-removed contrib/scale-test workflow; nothing in them applies to or is outstanding for contrib/perf-test.
harness.py read NUM_DEVICES/DEVICE_DIR/DEVICE_TAG from os.environ at module import, before Locust ever parses argv — so the --num-devices/--device-dir/ --device-tag CLI flags were silently dead code; only the env-var path (as set by docker-compose.yml) ever worked. Move resolution into an @events.init listener that reads environment.parsed_options once Locust has actually merged CLI args with env_var= defaults. Also parameterize _headers() to accept optional target/ostreehash overrides, needed by the upcoming update-check/download flow to report a real resolved target instead of the hardcoded placeholder.
The admin REST API (GET /v1/devices) is a structurally different actor from mTLS devices: separate host (:8080 vs :8443), Bearer-token auth instead of client certs, and its own low, fixed call cadence — so it gets its own module and User class rather than being bolted onto DeviceUser. Sized independently via --num-admins (fixed_count), so admin traffic never displaces device registrations by stealing virtual users from -u/ --num-devices. Locust's Runner.spawn_users() overwrites every registered User class's `host` attribute with the mTLS device host right before each spawn, so AdminUser can't rely on self.host/HttpSession.base_url — list_devices always builds absolute URLs instead, including when following the pagination Link header (which the server returns as a relative path).
Model the check-for-update + download sequence (timestamp -> snapshot -> targets -> resolve target/hash -> download-urls -> ostree file fetch) as a locust.SequentialTaskSet rather than independent weighted @task methods, since the steps are inherently ordered and later steps depend on state resolved by earlier ones (the real target name/hash parsed from targets.json). That state lives on the TaskSet instance, not the User or a module global, so it stays scoped to one device's flow. The flow is weighted via --update-flow-weight, 0 by default: only turn it on once a rollout has been seeded for these devices (see the upcoming gen-certs --seed-update), or every run 404s/400s on /repo/* and /ostree/*. Because PerfUser.tasks is normally built once at class-definition time by TaskSet's metaclass, making the weight configurable at runtime means rebuilding .tasks by hand inside the init listener, after CLI args have actually been parsed.
…ate) Adds an opt-in fixture-seeding step so Locust's check-for-update/download tasks have something real to hit: gen-certs now writes an unsigned-but- structurally-valid targets.json/snapshot.json/timestamp.json/root.json plus a real 256KiB ostree object, registers the update, and assigns it to every generated device via a rollout. Implemented directly at the storage layer (mirroring cmd/seed's pattern) rather than through the REST API's signed-tar-upload endpoint: the gateway never verifies TUF signatures on GET (that's a TUF-client-side concern), so unsigned fixture content is sufficient here and far cheaper to produce. Two correctness details that a naive port of cmd/seed's pattern would have gotten wrong: - Rollouts are assigned by UUID, not group. A device that self-registers via real mTLS always has group_name="" (DeviceCreate never sets it), so a rollout keyed by groups (as cmd/seed's seedUpdates() does) would silently match zero devices here. - Seeded devices get their real extracted pubkey, not a placeholder. Using a fake key would make the actual Locust mTLS handshake hit "Key rotation is not supported" on first contact, since the DB pubkey wouldn't match the cert gen-certs already wrote to disk for that same device. - gen-certs runs before "dg-sat serve" ever starts, so device rows don't exist yet and CommitRollout's tag-based match would have nothing to match against; seeding registers the devices and fakes a check-in to set their tag column before creating the rollout.
locustfile.py imports admin.PerfAdminUser, but the Dockerfile only ever copied setup.sh/harness.py/locustfile.py into /perf — admin.py was missing from the image, so the import silently failed and the admin actor never ran in any containerized invocation.
… setup.sh Plumbs the new gen-certs flags through setup.sh's own CLI args and matching env vars (SEED_UPDATE/UPDATE_TAG/UPDATE_NAME, default off), so the seeding step can be toggled from docker-compose/Makefile without editing the script.
Wires the new setup.sh/gen-certs seeding flags and the new admin/update- flow Locust knobs through docker-compose.yml's environment blocks and the Makefile's COMPOSE_ENV, all off/default by default so existing invocations (make run/setup/headless with no extra vars) behave exactly as before. DEVICE_TAG now derives from UPDATE_TAG (both default "main") rather than being hardcoded separately, since a mismatch between the two makes /repo/* 404 even with seeding enabled.
…veat Covers the admin actor, the check-for-update/download flow and its --update-flow-weight gate, how to seed a TUF target with SEED_UPDATE, and corrects the old blanket "/repo/* is intentionally excluded" note now that seeding makes it exercisable.
Modeled on test-dspl's hub_registry.py convention: every task in
locustfile.py and admin.py now carries @locust.tag(...) — device:check-in/
device:config/device:events on PerfUser, admin:list-devices on
PerfAdminUser, and update/update:check/update:download on UpdateFlow
(class-level @locust.tag("update") cascades to all five of its steps).
This lets --tags/--exclude-tags isolate one scenario at a time (ceteris
paribus) instead of steady-state/admin/update traffic all competing for
the same SQLite writer in one run.
Drops the custom --update-flow-weight CLI flag entirely in favor of
Locust's native tag filtering — PerfUser.tasks goes back to a plain
{callable: weight} dict (UpdateFlow included at weight 1), since the
weight no longer needs to be resolved at runtime from a parsed CLI option.
Tradeoff: a default/unseeded run now shows 404s on the update flow unless
--exclude-tags update is passed, whereas before it was silently excluded
by default; this is intentional — see README's "Running isolated
scenarios".
…tion Adds four convenience targets (locust-update-check, locust-update, locust-admin, locust-steady-state) so the common isolated scenarios don't require remembering the LOCUST_ARGS/--tags incantation. Also adds a profiles/scenes split modeled on test-dspl's scenes/profiles directories, reinvented for Make + Docker Compose instead of test-dspl's Helm-value-merging (there's no equivalent multi-key structural merge needed here — just NUM_DEVICES/SPAWN_RATE/RUN_TIME for profiles and LOCUST_ARGS for scenes, so plain `VAR := value` .mk files consumed via Make's own `include` are the natural fit). `make headless-scenario PROFILE=smoke SCENE=update-check` composes both; any value can still be overridden on the command line, since command-line assignments take precedence over both target-specific and included-file assignments (verified empirically with throwaway Makefiles before relying on it). Note for future extension: the four named targets set LOCUST_ARGS directly rather than SCENE, because the ifneq/include mechanism that resolves PROFILE=/SCENE= runs once at Make's parse time, before any target-specific variable assignment ever takes effect — documented in the README's Notes section.
…omposition Covers the new --tags/--exclude-tags workflow, the four named locust-* targets, and the profiles/scenes composition mechanism (with the command-line-override and PROFILE=/SCENE=-vs-target-specific-variable caveats). Also corrects the "Seeding a TUF target" example now that --update-flow-weight is gone.
Full-scale run against the current (tagged) harness, replacing the 2026-07-10 baseline. 810,439 requests, 0 failures, all 5000 devices registered. Confirms both previously-investigated latency patterns (SQLite single-writer contention on first-contact writes; timestamp.json's solitary long tail among UpdateFlow's siblings) are unchanged, since neither the tagging work nor the server's write path were touched by this session's changes. Includes two explicit correction notes for a units bug caught during review: five endpoint rows in the original table mislabeled sub-second millisecond max-latency values as seconds (a 1000x error), contradicting the doc's own prose two sections later. All nine rows are now independently verified against the raw stats CSV.
Every target already had a ## doc-comment; help just surfaces them, generated live from the Makefile itself and from what's actually on disk under profiles/ and scenes/, so it can't drift out of sync with what's implemented. Bare `make` (no target) now shows the same output via .DEFAULT_GOAL, since that's the most likely first command a newcomer runs. This was the one concrete, actionable friction point flagged by review that hadn't been addressed yet — PROFILE=/SCENE= were previously only discoverable by reading the README.
…running it Factors run/setup/headless's docker compose invocations into RUN_CMD/ SETUP_CMD/HEADLESS_CMD variables and gates each recipe on `ifdef DRY_RUN`, so `make locust-update-check NUM_DEVICES=20 SEED_UPDATE=1 DRY_RUN=1` prints exactly the one command that would run, with every variable/profile/scene already resolved — unlike `make -n`, which walks the whole prerequisite chain and echoes every target's recipe along the way. Propagates through headless-scenario and all four locust-* targets for free, since they're all just `headless` with LOCUST_ARGS preset.
Signed-off-by: Eric Bode <ebode@qti.qualcomm.com>
doanac
reviewed
Jul 13, 2026
| @@ -0,0 +1,190 @@ | |||
| // Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. | |||
Member
There was a problem hiding this comment.
When you move to the latest main, you'll see this: https://github.com/foundriesio/update-server/tree/main/cmd/seed I'm not sure if its worth consolidating code - but a single command to generate proper device certs isn't a bad idea. A customer could want this for manufacturing
Member
Author
|
Split it into 2 PRs. #211 and StealthyCoder#1 which will be retargeted against upstream main once #211 lands. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Implements a self-contained, fast
contrib/perf-test/Locust harness fordg-sat, replacingcontrib/scale-test. Registers/drives thousands ofsimulated devices over mTLS, plus an admin actor over the REST API, plus a
TUF check-for-update/download flow — with isolated, tag-selectable
scenarios rather than one monolithic mixed run.
Why a new folder instead of extending
scale-testcontrib/scale-test's cert generation forksopensslper device — at5000 devices that's tens of thousands of process spawns, minutes just to
get to a runnable state.
contrib/perf-test/gen-certsis a single Gobinary doing all keygen+signing in-process: 5000 ECDSA P-256 device certs
in ~250ms. Everything else (setup, seeding, the Locust file itself) is
also self-contained under one
docker compose/Makefileentry point.scale-testis removed in this PR rather than kept alongside it — twoparallel, easily-diverging perf-testing setups isn't worth the upkeep.
What it covers
:8443)GET /device,GET /config,POST /events,GET /repo/timestamp.json|snapshot.json|targets.json,POST /ostree/download-urls,GET /ostree/config:8080)GET /v1/devices(paginated)This is a subset of the server's full HTTP surface by design — see
"Follow-up" below.
Design notes
API.
gen-certs --seed-updatewrites an unsigned-but-structurally-validtargets.json/snapshot.json/timestamp.json/root, plus a real 256KiBostree object, and assigns it via a rollout — all before
dg-sat serveever starts. The gateway doesn't verify TUF signatures on
GET(that's aTUF-client concern), so unsigned content is legitimate here and far
cheaper than driving a real signed tar upload through the REST API for
zero benefit on the paths this harness actually exercises.
via real mTLS always has
group_name=""(DeviceCreatenever sets it) —a rollout keyed by
groupswould silently match nothing. Confirmed byreading
storage/gateway/storage.gobefore relying on it.fake key would make the real mTLS handshake hit "key rotation not
supported" on first contact, since the DB pubkey wouldn't match the cert
gen-certsalready wrote to disk for that device.self.host. Locust's runneroverwrites every registered
Userclass'shostwith the mTLS devicehost immediately before each spawn (
runners.py, confirmed by readingLocust 2.43.4's source, not assumed from docs) —
AdminUsercan't relyon
HttpSession.base_urland builds full URLs instead, including whenfollowing the paginated
Linkheader (server returns it as a relativepath, so it must be re-anchored per page).
device:*/admin:*/update:*,update:check/update:download), mirroring the@locust.tag(...)convention already used elsewhere for Locust harnesses in this org, so
--tags/--exclude-tagscan isolate one scenario at a time —ceteris paribus, without steady-state/admin/update traffic allcompeting for the same SQLite writer in the same run. This replaced an
earlier custom
--update-flow-weightCLI flag that only gated the wholeflow on/off; native tag filtering is strictly more capable and one less
thing to maintain.
make locust-*targets +PROFILE=/SCENE=composition.make locust-update-check,locust-update,locust-admin,locust-steady-statecover the common cases without needing to rememberthe
--tagsincantation;profiles/*.mk/scenes/*.mklet scale andtask-selection presets be composed (
make headless-scenario PROFILE=smoke SCENE=update-check), reusable by name, still overridable on the commandline (command-line assignments win over both target-specific and
included-file assignments — verified against GNU Make's actual
precedence rules, not assumed).
make help/ baremakelists every target from the##doc-comments already on them, plus whatever's actually on disk under
profiles//scenes/— can't drift out of sync with what's implemented,unlike a hand-maintained list would.
DRY_RUN=1prints the fully-resolveddocker composecommand insteadof running it — narrower than
make -n, which walks the wholeprerequisite chain and echoes every target's recipe along the way.
Under a device cold-start burst,
dg-sat's single-writer SQLite(
db.SetMaxOpenConns(1)) serializes first-contact writes; a small numberof unlucky requests queue before finally succeeding (e.g. ~240s tail at
our 80/s spawn cap, 5000 devices). Zero failures either way — it's a
latency problem, not a correctness one.
A follow-up spawn-rate scan (2000 devices, steady-state traffic only,
rates 10/20/40/80/160) found the tail has a clear knee: median/p95 latency
stay flat (10ms/~25ms) at every rate tested, but max latency grows
roughly linearly from 10→20/s then jumps nonlinearly from 20/s onward
(741ms → 1.4s → 6.9s → 26.5s → 39.8s), still with zero failures at every
rate including 160/s. The
Makefile's 80/s default sits well past thatknee. Full methodology, caveats, and reproduction steps in
contrib/perf-test/baselines/ratescan-2026-07-13.md; root-cause writeupin
contrib/perf-test/baselines/baseline-2026-07-11.md.Follow-up
The harness currently covers a fraction of the server's REST surface
(missing: apps/registry download, device hardware/network/apps-state
reporting, fio-tests, and the whole admin configs/rollouts/TUF-management
domain). Planned as a sequence of small follow-up PRs — one domain at a
time, each waiting on review of the previous before starting — rather than
one large diff. Tracked separately, not part of this PR's diff.
Testing
seeded + update-flow + admin mixed run): 0 failures, 0 exceptions, full
device registration confirmed via the
Linkheader'slastoffset.Results and analysis in
contrib/perf-test/baselines/.maketarget andPROFILE=/SCENE=combinationdry-run-verified (
make -n) and then actually run against a liveseeded server at small scale before being trusted at full scale.
make headless/make run(no extra flags) re-verified unaffectedafter each change that touched shared plumbing (config resolution,
tagging, the Makefile rewrite) — regression-checked, not just assumed
from reading the diff.