v1.4.0

After the 2,000,000 run, the question I kept getting was: what about a real
database? So I load tested a chat app that keeps every message in Postgres,
with 100 million messages in it, and it held 40,000 users on one 16-core
server, with Postgres on a machine of its own. That test is why the probe now
watches your database, and why a --find-limit verdict tells you which part
gave out. Runs also show up where you already work now, in your app's
LiveDashboard and in Livebook.

• --probe watches your database too. It sends how long queries waited for
  a connection from the Ecto pool every second, and for a Postgres repo it
  reads Postgres's own commits, writes, reads, connections and waits, over a
  connection of its own outside your pool. The dashboard and the HTML report
  chart them
• A --find-limit verdict says why the step failed when the probe can tell:
  queries waiting on the pool, what Postgres was waiting on, such as writing
  its log to disk, and whether the app's own schedulers were the limit
• A Stampede page for Phoenix LiveDashboard, Stampede.LiveDashboard. Add
  it to your app's dashboard, and a run that probes the app shows in a
  Stampede tab while it runs, next to the dashboard's own pages. Finished
  runs stay there, with the same results as the HTML report
• Load tests in Livebook. With Kino in the notebook, a Load test smart
  cell writes and runs a scenario for a page, a LiveView, a channel or an AI
  chat API, and can probe your app's node. Stampede.Kino.run/2 shows any
  run in the notebook while it goes, and any Stampede.Result shows as the
  report
• The bench has a new result: a chat app that keeps every message in
  Postgres, seeded with 100 million messages, held 40,000 users with no
  errors on one 16-core server, with Postgres on an 8-core machine of its
  own

The full test, with the chat app, the scripts and every log, is in
bench/million.

v1.3.2

The README has the newest result: one 16-core Phoenix server held 2,000,000 users with no errors.

v1.3.1

The README's images and links now work on hex.pm, and it no longer says that signing in isn't recorded.

v1.3.0

This one is about real apps. You can record yourself using your app and get a
scenario, signing in included, and see inside the app during a run without
changing it. Stampede can also load test AI chat APIs now, and it found new limits
for me: stock Phoenix held 600,000 Channels users on one 4-core server, and a real
open source Phoenix product held 50,000 users for 10 minutes on the same size of
machine. Everything needed to run those again is in bench/million.

• mix stampede.record starts your app and records what you do in the
  browser, then writes it as a scenario, with your clicks, forms, uploads,
  channels, navigation and pauses, and signing in through a form. Passwords
  and other sensitive fields become "CHANGE_ME", in forms, in JSON and in
  URLs. It's new, so it may still change
• Stampede.HTTP keeps cookies with cookies: true, one set per VU and per
  host, and Stampede.HTTP.clear_cookies/0 forgets them. :cookies on
  Stampede.LiveView.connect/2 and Stampede.SSE shares them, so a scenario
  can sign in over HTTP and then open a LiveView as that user
• Stampede.HTTP's :form option sends a form as a browser would, and
  Stampede.HTTP.csrf_token/1 reads the CSRF token out of a page
• A guide to watching a run in Prometheus and Grafana
• Stampede.Channels.connect/2 takes a :keep function, for VUs in busy
  channels that hear far more than they read
• --probe app@host, or probe: from code, watches the app from the inside
  during a run, with no changes to it: scheduler use, memory, processes, run
  queue and the longest message queues, charted in the dashboard and the
  HTML report, and Phoenix, LiveView and Ecto timings as server.* metrics.
  A --find-limit verdict says when the server's schedulers ran out, or
  when it went down
• Stampede.LLM load tests AI chat APIs in OpenAI's format, such as Ollama,
  vLLM or llama.cpp. It records time to first token, the gaps between
  tokens, time per token and tokens per second, measured the same way as
  vllm bench serve. It's new, so it may still change
• Stampede.LLM.prompts_from_jsonl/2 reads conversations from benchmark
  datasets, and Stampede.LLM.synthetic_prompt/1 makes prompts of a given
  length
• Stampede.SSE reads Server-Sent Events from any server, with when each
  event arrived
• mix stampede.gen.scenario --llm generates a scenario for a model server
• The dashboard and the HTML report show tokens per second and time to
  first token for runs that chat with a model server

v1.2.0

This one answers the question I get asked most: how many users can my app
handle? It also comes out of a real test on AWS, where stock Phoenix on one
4-core server held 400,000 Channels users with no errors. That run found
the problems with Stampede itself fixed below, and everything needed to run
it again is in bench/million.

• --find-limit, or find_limit: from code, answers "how many users can
  it handle?": the load goes up a step at a time until a threshold fails,
  and the result says the most users, or the highest rate, that passed.
  It's new, so its options and report may still change
• --badge, and Stampede.Reporter.to_badge/1, write a README badge with
  the limit found
• mix stampede.gen.scenario --rate generates a scenario that starts
  iterations at a rate
• mix stampede.compare --format markdown, to post a comparison on a pull
  request
• Finished runs on the dashboard can be downloaded as the HTML report or
  the JSON
• Channels and WebSocket connections hibernate after 5 seconds without
  traffic. A VU holding a Channels connection went from about 37 KB of
  memory to 23 KB, which matters once you're holding hundreds of thousands
• WebSocket connections use a 64 KB socket buffer instead of the few
  megabytes Mint sets. With 60,000 connections, that cut the load test's
  CPU by about a fifth
• Runs measure how busy the load test itself is, every second. When it's
  90% busy or more, its own delays end up in its timings, so the summary
  and the HTML report say so, and a --find-limit verdict says the limit
  may be the load test's rather than the server's
• A run ends on time and prints its summary even when its VUs keep the
  machine 100% busy. Its own processes run at high priority, and progress
  is printed without holding the run up
• A VU whose iterations keep failing waits longer between tries: 100 ms,
  then 150, 200, 250, 500 ms, 1 s, 2 s and at most 5 s, much like Phoenix's
  JavaScript client before it reconnects. It was 100 ms every time, which
  piled more load onto a server, or a load test, that was already in
  trouble
• Connections no longer go through Erlang's host lookup for every
  connect. IP addresses are used as they are, and host names are looked
  up once and kept for 5 minutes. Under heavy load, lookups queued behind
  each other and timed out as nxdomain errors that looked like the
  server failing
• A run warns before it starts when a node may not allow enough processes
  or open files for its VUs, instead of failing halfway with errors that
  don't say why. It checks ulimit -n too, which the BEAM doesn't report

v1.1.0

This release is mostly about seeing what a load test is doing while it runs,
and about covering more of a Phoenix app.

• A live dashboard: mix stampede.run --dashboard, or Stampede.Dashboard
  from code
• --parallel and Stampede.run_parallel/2 run several scenarios at once,
  each with its own options
• LiveView uploads with Stampede.LiveView.upload/4, recorded as
  "live.upload"
• mix stampede.gen.scenario generates HTTP, Channels, LiveView and
  WebSocket scenarios
• mix igniter.install stampede adds Stampede and a first scenario to a
  project
• Stampede.ExUnit runs load tests from a test suite
• Binary payloads in Stampede.Channels with the "2.0.0" serializer, as
  {:binary, data}
• Custom serializers through Stampede.Channels.Serializer and the
  :serializer option, for servers that extend the protocol
• Elixir 1.18 is supported
• New guides: pages behind a login, Phoenix Presence, Absinthe
  subscriptions, load tests in CI, spreading the load over machines, and a
  case study with real numbers

Fixes

• A payload that cannot be encoded raises in the caller of push/5 or
  cast/4, instead of crashing the socket.

v1.0.0

First release of Stampede, load testing for Phoenix with built-in Channels and LiveView support.

• LiveView: loads pages like a browser and sends clicks, form changes, submits, patches and navigation
• Phoenix Channels, HTTP and plain WebSockets
• VUs with stages, or a fixed arrival rate, with graceful stop
• Checks and thresholds that fail CI, JSON and HTML reports, and mix stampede.compare for regressions
• One test spread over several nodes

See the README to get started.