Skip to content

feat(perf-test): self-contained mTLS Locust harness + real per-device seed certs#211

Merged
StealthyCoder merged 2 commits into
foundriesio:mainfrom
StealthyCoder:perf-test-pr1
Jul 16, 2026
Merged

feat(perf-test): self-contained mTLS Locust harness + real per-device seed certs#211
StealthyCoder merged 2 commits into
foundriesio:mainfrom
StealthyCoder:perf-test-pr1

Conversation

@StealthyCoder

Copy link
Copy Markdown
Member

What

Adds contrib/perf-test/, a self-contained Locust-based mTLS performance
test for dg-sat: a fast in-process Go cert generator (replacing
contrib/scale-test's openssl-fork-per-device approach), a device actor
(mTLS :8443, GET /device), and an admin actor (Bearer token :8080,
paginated GET /v1/devices). This is deliberately scoped down — /config,
/events, the update-check/download flow, and task tagging/named Makefile
targets are follow-up work in a stacked PR (see below), so this stays
reviewable as a first increment.

Also: contrib/scale-test (the superseded Python/openssl harness) is
removed, and cmd/seed is updated to generate real per-device certs instead
of a shared dummy pubkey, per review feedback that the two shouldn't
duplicate cert-generation logic.

Why a new folder instead of extending scale-test

contrib/scale-test's cert generation forks openssl per device — at scale
that's thousands of process spawns, minutes just to get to a runnable state.
contrib/perf-test/gen-certs is a single Go binary doing all keygen+signing
in-process: 5000 ECDSA P-256 device certs in ~250ms.

What it covers

Actor Endpoints
Device (mTLS :8443) GET /device
Admin (REST :8080) GET /v1/devices (paginated)

cmd/seed now shares cert generation with gen-certs

cmd/seed's seedDevices() used to register every seeded device with one
hardcoded dummy RSA pubkey — fine for populating the UI, but nothing that
could ever mTLS-authenticate. This PR extracts gen-certs's CA/leaf
gen-certs and cmd/seed call it:

  • cmd/seed now issues each device a real ECDSA leaf cert from a CA
    generated lazily on first use, appended to cas.pem (not overwritten, so
    contrib/run-local.sh's own dev TLS leaf-as-CA keeps working alongside
    it). Cert+key land under datadir/certs/seed-devices/<uuid>/client.pem.
  • Device UUIDs stay on the existing deterministic seed-device-%05d scheme
    — only the pubkey/cert material changed, not device identity.
  • Re-running seedDevices against an existing datadir skips cert
    (re)generation for devices that already have one on disk, so it doesn't
    silently invalidate a cert still valid in the DB. Covered by
    TestSeedDevicesIdempotent.

Design notes

  • Admin traffic uses absolute URLs, not self.host. Locust's runner
    overwrites every registered User class's host with the mTLS device
    host immediately before each spawn — AdminUser builds full URLs
    instead, including when following the paginated Link header (the
    server returns it as a relative path).
  • storage/devcerts is crypto-only, no file I/O — callers keep full
    control over on-disk layout and concurrency (gen-certs's worker pool
    over runtime.NumCPU() is unchanged).

Testing

  • go build ./..., go test ./..., go vet ./..., gofmt -l . all clean.
  • storage/devcerts has its own tests: CA generation, leaf issuance,
    signature verification against the issuing CA.
  • cmd/seed's TestSeed extended to assert seeded devices get distinct
    real pubkeys (not a shared dummy), a real cert file is written, and the
    cert chains to the CA appended in cas.pem. New TestSeedDevicesIdempotent
    covers re-running against an existing datadir.
  • Manually ran ./contrib/run-local.sh end-to-end: confirmed cas.pem
    contains both the original dev TLS leaf and the new seed CA, and a seeded
    device's cert verifies against it via openssl verify.
  • contrib/perf-test: built gen-certs, ran it standalone (device certs
    verify against cas.pem), and confirmed docker compose config parses.

Follow-up

A stacked PR (targeting this branch) adds /config, /events, the
check-for-update/download flow, per-task tagging, and Makefile/profile/scene
ergonomics. Kept separate per review feedback that the original combined PR
was too large to review as one unit.

@doanac doanac left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

None of your commits have signed-off-by or co-authored-by.

Comment thread contrib/README.md
Comment thread contrib/perf-test/docker-compose.yml Outdated
Comment thread contrib/perf-test/Makefile
Comment thread cmd/seed/main.go Outdated
StealthyCoder and others added 2 commits July 16, 2026 10:43
…o cert generator

Existing openssl-fork based cert generation takes minutes for thousands
of devices; a single in-process Go generator does 5000 ECDSA certs in
~250ms. Verified end-to-end (10 and 5000 devices): golang:1.24 builder
bumped to 1.25 to match go.mod, setup.sh runs auth-init before tuf-init
(TUF needs the HMAC secret auth-init creates), and the locust service's
/data mount is read-write so headless runs can write their CSV/HTML
report.

Names the built binary and hostname "fioserver" throughout, matching
cmd/server's actual binary name (see contrib/server/Dockerfile,
contrib/run-local.sh) rather than an earlier prototype's "dg-sat".

Signed-off-by: Eric Bode <ebode@qti.qualcomm.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The admin REST API (GET /v1/devices) is a structurally different actor
from mTLS devices: separate host (:8080 vs :8443), Bearer-token auth
instead of client certs, and its own low, fixed call cadence — so it gets
its own module and User class rather than being bolted onto DeviceUser.

Sized independently via --num-admins (fixed_count), so admin traffic never
displaces device registrations by stealing virtual users from -u/
--num-devices.

Locust's Runner.spawn_users() overwrites every registered User class's
`host` attribute with the mTLS device host right before each spawn, so
AdminUser can't rely on self.host/HttpSession.base_url — list_devices
always builds absolute URLs instead, including when following the
pagination Link header (which the server returns as a relative path).

Signed-off-by: Eric Bode <ebode@qti.qualcomm.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@StealthyCoder
StealthyCoder merged commit 70cd0b0 into foundriesio:main Jul 16, 2026
14 checks passed
StealthyCoder added a commit to StealthyCoder/update-server that referenced this pull request Jul 21, 2026
setup.sh calls `fioserver user-add --tokenfile ...` to seed an admin
API token before the server starts, but UserAddCmd never grew that
flag — it only existed on an unrelated, unmerged perf-test-idea
branch. Add it back: after creating the user, generate a 30-day token
scoped to the requested roles and write its value to the given file.

Reported against PR foundriesio#212's stacked setup.sh, but the call was actually
introduced in PR foundriesio#211 (merged into main), so this targets that gap
directly rather than rebasing merged history.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
StealthyCoder added a commit to StealthyCoder/update-server that referenced this pull request Jul 21, 2026
setup.sh calls `fioserver user-add --tokenfile ...` to seed an admin
API token before the server starts, but UserAddCmd never grew that
flag — it only existed on an unrelated, unmerged perf-test-idea
branch. Add it back: after creating the user, generate a 30-day token
scoped to the requested roles and write its value to the given file.

Reported against PR foundriesio#212's stacked setup.sh, but the call was actually
introduced in PR foundriesio#211 (merged into main), so this targets that gap
directly rather than rebasing merged history.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants