Skip to content

ProjectAchilles v2.9.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 02 Nov 03:07
· 108 commits to main since this release
v2.9.0
0eaf21f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Performance & Scale. This release is about growing agent fleets on small instances. On a backend capped at 0.5 CPU / 512 MB (the smallest hosted instance size), 3,000 simulated agents run at p95 3 ms, and restarting the backend with all of them connected marks none offline; before, agents started dropping offline after every deploy at about 110 agents. On live installs, an agent poll went from 728 ms to 2–3 ms, selecting a run in Executions from three requests of 1.2–1.8 s to one of 370 ms, and the Secure Score trend loads ~80× less data. Also in this release: SQLite retention, paced test-binary handouts, Elasticsearch indices that stay green on single-node clusters, rate limits that no longer throttle agents behind one NAT, and CrowdStrike and Sophos alert correlation through process lineage. Reproduce the capacity numbers on your own hardware with backend/scripts/load/agent-fleet-sim.ts.

Upgrade notes

  • First start after upgrading: each agent's next request runs one bcrypt check to record its key digest; after that, restarts cost nothing. Small fleets finish in seconds; about 1,000 agents on half a CPU take ~6 minutes, during which some may briefly show offline. Upgrade large fleets off-hours.
  • Existing Elasticsearch indices keep their replica setting. Run the one-time _settings command under Fixed to turn a single-node cluster green.
  • Daily write-index rollover creates an index per testing day. On small or shared clusters choose Monthly or None in Analytics settings.
  • New optional settings (safe defaults): TASK_OUTPUT_RETENTION_DAYS (90), AGENT_EVENT_RETENTION_DAYS (90), AGENT_DOWNLOAD_BUDGET_MB_PER_MIN (1500), ES_TIMING_LOG.
  • Known limitation: Update All agent updates aren't paced yet (#703); on large fleets, update in batches.
  • No agent release is needed; agents stay on 0.7.2. Rolling back to 2.8.0 keeps agents authenticating: their bcrypt key hashes are untouched.

Added

  • Agent fleet load test. backend/scripts/load/agent-fleet-sim.ts simulates N agents at the real poll and heartbeat cadence against a backend and reports per-minute latency, errors and agents silent past the 180 s offline threshold. --seed-only seeds a fresh database and refuses ~/.projectachilles or any database holding real agents. The README has the measured results and how to cap the backend like a small hosted instance. (#697)

  • Elasticsearch time per API request. Every API response that queried Elasticsearch carries a Server-Timing header (number of calls, time spent waiting for them, and the took Elasticsearch reports), and the backend logs one [es-timing] line for it. The gap between the wait and took is network and connection setup, which tells a slow query apart from a distant cluster. Set ES_TIMING_LOG=off to drop the log line. Both backends.

  • Old task output and agent events are pruned. The SQLite database grew without bound: every task result keeps its full stdout (about 15 KB each) and every agent logs about ten events a day. Hourly, finished tasks older than TASK_OUTPUT_RETENTION_DAYS (default 90, 0 disables) lose their stdout, keep the last 400 characters of stderr, and lose each stage's details. Test results are compacted only once they are in Elasticsearch; agent update and uninstall output, which never goes to Elasticsearch, is compacted after the window too; admin shell-command (execute_command) output is never compacted, because it is audit-relevant. The exit code, error, EDR-kill marker and per-stage results stay, so verdicts and the campaign matrix do not change, and the Tasks page notes when the output was removed. Agent events older than AGENT_EVENT_RETENTION_DAYS (default 90) are deleted, except each agent's latest went-offline event, so an agent that has been offline longer than the window is not reported as going offline again. A failure in one job no longer skips the other. The file does not shrink, but freed pages are reused. Backend only.

  • Test binary handouts are paced. When a schedule or Run Now targets many agents, every agent used to download its test binary within one poll window, a burst that could exceed the agent's download timeout on a small instance. The backend now hands out test binaries at a per-minute budget (AGENT_DOWNLOAD_BUDGET_MB_PER_MIN, default 1500 MB, 0 = unlimited); agents over the budget get "no task" and retry at their next poll. Agent update tasks are not paced. At the default, a 30-minute pre-execution notice pickup window covers about 2,600 agents with a 17 MB binary (about 1,100 with 40 MB); raise the budget or the window for more. Backend only.

Security

  • Failed agent authentication is limited per IP. A request with a random ak_ key and a made-up X-Agent-ID costs the backend one bcrypt cost-12 compare (about 0.2–0.5 s of main-thread CPU), and nothing limited such requests per IP: the agent device limiter is keyed on the client-chosen agent ID, so changing it on every request escaped it, and the CLI and UI limiters skip agent device routes. The backend now counts, per IP, authentication failures from agent IDs it doesn't know, at the moment it finds the ID is unknown; after 60 in a minute, further agent requests from that IP get 429 "Too many failed agent authentication attempts" (with Retry-After) before any bcrypt work, until the minute is over. Nothing else counts: successful requests, requests still in flight (such as long binary downloads) and failures by enrolled agents (decommissioned, stale key, clock skew; already capped per agent) never do, so agents behind one NAT aren't throttled by their neighbours. The 401 response is unchanged. Backend only.
  • Case-variant admin paths no longer skip the dashboard rate limit. The global UI limiter skipped agent device routes by a case-sensitive path check, so a Clerk session calling /api/agent/ADMIN/... (Express routing is case-insensitive) bypassed its per-user budget. The UI and CLI limiters now share one case-insensitive device-route check. Backend only.
  • Agent API keys are no longer sent to Clerk. Clerk's Express middleware treats a Bearer token that starts with ak_ (Clerk's own API-key prefix) as a Clerk API key and verifies it with POST api.clerk.com/api_keys/verify, and it can't be configured not to. Agent keys use the same prefix, so every agent poll and heartbeat sent the agent's key to Clerk, adding ~140 ms each. Clerk now only sees JWT-shaped Bearer tokens (session and CLI tokens); for any other Authorization header the request is signed out to Clerk and the header is passed on unchanged to the agent / API-key authentication. Both backends.

Fixed

  • Agents behind one IP are no longer throttled by the CLI bearer rate limiter. Agent requests (Bearer ak_…) counted against the 60 requests/minute per-IP budget meant for CLI tokens, so more than about 20 agents behind one office NAT, proxy or VPN got 429 on heartbeat and poll and were marked offline after 180 s. Agent device routes (/api/agent/* except /api/agent/admin/*) now skip that limiter; they keep their own per-IP-and-agent limiter, and ak_ tokens on any other path are still counted. Backend only.

  • New Elasticsearch indices no longer leave an unassignable replica on single-node clusters. Achilles created its indices without a replica setting, so Elasticsearch used its default of 1 replica; on a one-node cluster that replica can never be assigned (the cluster stays yellow, and the shard count and heap grow with every index). All Achilles indices (results, archive, Defender, CrowdStrike, SentinelOne, Sophos, risk acceptance) are now created with index.auto_expand_replicas: 0-1: no replica on one node, one replica when a second data node exists. Clusters that reject the setting (Elastic Cloud Serverless) get the index without it. The Write index rollover setting now warns that Daily costs memory on small clusters. Both backends. Indices that already exist are not changed by this release; to fix them, run this once in Kibana Dev Tools:

    PUT achilles-*,archived-achilles*/_settings
{"index.auto_expand_replicas":"0-1"}

    or with curl:

    curl -X PUT "$ES_URL/achilles-*,archived-achilles*/_settings" \
  -H "Authorization: ApiKey $ES_API_KEY" -H 'Content-Type: application/json' \
  -d '{"index.auto_expand_replicas":"0-1"}'

  • Auto-resolve says why a pass failed. When SentinelOne refused to resolve alerts (HTTP 403: the API token's user had the Viewer role, which can't change incident status), the backend logged only errors=1 and the settings card showed nothing. The Defender, CrowdStrike, SentinelOne and Sophos auto-resolve cards now show the last pass's errors: the counts, the first error and how many more there are. SentinelOne and Sophos also log the errors (up to five per pass) next to the count, as Defender and CrowdStrike already did.

  • Sophos XDR detections raised on a process a test started now correlate. A detection's main entity list names only the process it fired on (cmd.exe, powershell.exe, rundll32.exe). The test's stage binary that started it is listed in the detection's source_entities, which the sync didn't request. On a live tenant, a blocked LNK, a headless-conhost launch and a rundll32 CPL launch matched no stage. The sync now reads the File source entities into the evidence, and stage attribution takes the stage binary. The Sophos sync version moves to 2, so the next poll clears the checkpoints once and backfills, and existing detections correlate on the following pass. The SIEM↔XDR merge is unchanged on the live capture.

  • One malformed stage no longer fails the whole related-alerts batch. POST …/correlation/alerts-for-tests rejected the entire Executions row with a 400 when any stage failed validation (how the #678 long-label bug blanked every stage). Each stage is now validated on its own: an invalid one comes back as null with its reason in errors, and the other stages are still answered. A missing or empty items list, or more than 100 items, is still a 400.

  • A failed related-alerts lookup no longer reads as "No related alerts". When the alert request for a run failed (an HTTP error, for example), the run drawer showed the same text as a run with no alerts, and the failure was remembered until the page was reloaded. The drawer now says "Couldn't load alerts. Select the run again to retry." in the vendor's section and in a stage's EDR evidence. Only successful lookups are kept, so selecting the run or stage again asks again.

  • The Secure Score trend loads ~80× less data. Each daily Secure Score document also stores every control's score, and the trend query (Dashboard, Analytics, the Defender tab and the Defense vs Secure Score chart) fetched whole documents: 588 KB for 30 days on a live tenant, against 7 KB for the four fields it uses. With the cluster far from the backend that transfer took over a second. Both Secure Score readers now request only their fields. Both backends.

  • CrowdStrike auto-resolve closes alerts again. Every PATCH /alerts/entities/alerts/v3 failed with HTTP 400 "failed to read and parse request", because the status and comment were sent as top-level update_status / append_comment keys. The v3 API takes them as action_parameters: [{ name, value }] (CrowdStrike's Swagger reference and FalconPy). Nothing was ever closed in Falcon while the mode was Enabled. Once you upgrade, the next pass closes up to 30 correlated alerts, and later passes work through the rest. Both backends.

  • Runs whose test lists long-form technique labels show their EDR alerts again. The new batch related-alerts route rejected any technique longer than 50 characters, and some tests list labels like T1562.001 (Impair Defenses: Disable or Modify Tools) (52). One such label returned 400 for the whole run, so every EDR section read "No related … alerts" even when alerts were correlated. Techniques now get the same 300-character limit as the other fields.

  • Selecting a run in Executions asks each EDR once, not once per stage. The related-alerts panel fetched every stage of the selected run separately for every configured EDR (Defender, CrowdStrike, SentinelOne, Sophos), each lookup up to three searches in a row: about 80 requests for a 40-control bundle with two EDRs. A new POST …/correlation/alerts-for-tests per vendor answers all stages with one Elasticsearch msearch per fallback tier, and the lookup no longer repeats its technique-plus-host search as a third tier (same query, always the same empty answer). The console falls back to per-stage requests on backends without the new route.

  • The Analytics filter dropdowns load with one request. The page asked ten available-* endpoints for its filter lists, each a separate Elasticsearch query; with the cluster far from the backend they all paid the round trip and most needed a new connection. GET /api/analytics/filter-options returns all ten lists from one query (the old endpoints still work). Both backends.

  • CrowdStrike alerts raised on a process a test started now correlate. Tests act through Windows binaries, so most behavioural Falcon alerts fire on cmd.exe, powershell.exe or conhost.exe. The test binary carrying the test UUID is that process's parent or grandparent. The sync kept only the triggering process, so those alerts matched no test. It now also stores the parent and grandparent file names and paths that Falcon sends with every alert. Stage attribution takes the nearest stage binary. The alert sync version moves to 3, so the next poll re-reads the last 90 days once and existing alerts correlate on the following enrichment pass. Both backends.

  • A standalone test's run drawer shows its EDR alerts. The drawer listed only stage-attributed alerts, and a standalone test has no stages and no bundle-level section, so a correlated alert (the test binary's own) was hidden behind "No stage-specific … alerts. See the bundle-level alerts above." Standalone runs now list every correlated Defender, CrowdStrike, SentinelOne and Sophos alert.

  • The Analytics dashboard loads its data once, and the EDR cards no longer wait for it. Each EDR's "configured" check answers just after the page opens, and that re-ran all 13 dashboard requests, so every load queried Elasticsearch twice. The Defender, CrowdStrike, SentinelOne and Sophos summaries now load on their own, in parallel with the dashboard instead of one after another once it finishes, and only reload when the date range changes, not on every filter change.

  • Agent requests no longer cost a bcrypt check each, even after a restart. The backend remembered a verified agent key in memory for only 30 seconds, the agent's poll interval, so nearly every GET /api/agent/tasks re-ran a cost-12 bcrypt compare: about 0.7 s on a half-CPU instance, in slices that held up every other request for up to ~100 ms. After each restart or deploy every agent's next request ran one at the same moment, and a few hundred agents could hold up the event loop long enough to be marked offline. The backend now stores an HMAC-SHA256 digest of each proven agent key in a new agent_key_digests table (with a write-through copy in memory), keyed by and bound to the key hash it was checked against, so a poll needs no bcrypt and a restart costs one SQLite read per agent. A newly enrolled agent's digest is recorded at enrollment, and requests that arrive together with the same key share one bcrypt compare. The digest is never used while a key rotation is pending, and a replaced key or a deactivated agent still takes effect within 30 seconds. agents.api_key_hash is unchanged, so rolling back to an earlier release is safe.

  • A section that kept failing to load in one browser now repairs itself. nginx sent the year-long immutable cache header on 404 responses too, so a browser that once asked for a code chunk an instance did not have (while a deploy switched instances) kept that 404 and never asked again: every page importing the chunk failed in that browser, reloads included. Missing chunks now get a no-store 404, and before reloading, the console re-downloads any cached chunk in the failed page's imports that isn't JavaScript, which repairs browsers already affected. On Vercel, a missing chunk returns 404 instead of the app's index.html.

  • A failed page load after a deploy no longer leaves the console on an error page. When a section's code chunk fails to load, the console reloads itself to pick up the current build. It allowed one reload per 15 seconds, so a second failure right after the reload stopped on "Something went wrong", and its Try Again button could not recover (the failed import is kept until the page reloads). The console now reloads up to twice a minute, and the error page's button for this error is Reload page, which does a full reload.

  • No more error-page flash while the console updates itself. During that automatic reload the console briefly showed "Something went wrong: Cannot read properties of undefined (reading 'default')" until the new page arrived, most visibly on hosted installs after each frontend deploy. It now shows "Updating to the latest version…" until the reload finishes.

  • The Defender tab's top-controls card makes one request instead of ten. It asked /controls/correlation once per listed control, so every Defender tab load sent ten requests in the same second on top of the other tiles. It now calls the new GET /api/analytics/defender/controls/correlations (up to 20 controls, one Elasticsearch filters query with the same clauses, so each count is unchanged). The single-control route stays. Both backends.

  • The EDR analytics tabs (Defender, CrowdStrike, SentinelOne, Sophos) reuse one Elasticsearch connection. Every request on those tabs built a new Elasticsearch client, so each one paid a DNS lookup, TCP connect and TLS handshake (about 165 ms to Elastic Cloud from a distant site, against 1–3 ms for the query itself) and stalled for seconds whenever the network dropped one; the clients were never closed. They now share one kept-open client, rebuilt when the analytics connection settings change (the old one is closed a minute later). The auto-resolve status and receipts routes, posture tiles and the campaign/notice EDR lookups use it too. (#668)

  • A brief DNS or connection failure no longer fails a whole Sophos sync. The Sophos client retried HTTP 429 and 5xx responses but not network errors, so one slow DNS answer for id.sophos.com or api.central.sophos.com failed the poll with "whoami: fetch failed". Failures before the request is sent (DNS, refused, connect timeout) are now retried on the same backoff for every method; a connection dropped mid-request is retried only for reads. The final error names the host. (#667)

  • "Sophos sync failed" clears once a later sync succeeds. An error from the steps before the alert, SIEM and XDR pulls (such as whoami) stayed on the Sophos tab until the backend restarted, however many syncs succeeded after it. SentinelOne had the same problem when its threats source was turned off. (#667)

  • Network errors say what failed. The Sophos and SentinelOne sync status showed only "fetch failed"; it now adds the system error, for example fetch failed (EAI_AGAIN: getaddrinfo EAI_AGAIN id.sophos.com) for a DNS timeout or SELF_SIGNED_CERT_IN_CHAIN for TLS inspection. (#667)

Changed

  • Render can be deployed from GitHub Actions. Render's "After CI checks pass" auto-deploy waits on the commit's combined status, so an unrelated failing status (a rate-limited Vercel preview) held four merges off a hosted install with no error shown. .github/workflows/deploy-render.yml deploys each Render service through its deploy hook once CI passes on main, pinned to the tested commit and limited to services whose files changed. Opt-in: add the RENDER_DEPLOY_HOOK_BACKEND / RENDER_DEPLOY_HOOK_FRONTEND secrets and turn Render's auto-deploy off; without them it does nothing. See the Render guide.
  • The execution report names the organization from the console identity. The cover, the scope table and the page headers print the full name saved in Settings → General, then CONSOLE_ORG_NAME, then ORGANIZATION_NAME, so a name set in the console no longer needs a redeploy to reach the PDF. A saved identity replaces the environment values, as it does in the console. Names are still cut to 56 characters in the report. backend-serverless/, which has no console identity, keeps reading ORGANIZATION_NAME. (#666)
Assets 7
Loading