Goose 0.19.0

Summary

Goose 0.19 adds a live web dashboard that shows a load test while it runs and can steer it, and ships a rebuilt Goose Book that documents every option, report and Controller against what Goose actually prints. Writing that Book turned up dozens of defects in metrics, Controllers and option parsing, and 0.19 fixes them, so several numbers and refusals differ from 0.18.

Highlights

• Live dashboard: --dashboard serves a browser view of a running load test, with graphs, request, scenario and transaction tables, a response time heatmap and coordinated omission figures; --dashboard-control adds start, stop and ramp controls. Live Dashboard
• Thresholds: --threshold rules such as p95<500ms pass or fail a run, and a failed rule makes execute() return an error so the process exits nonzero. Thresholds
• One folder per run: output flags given without a path write into a folder for the run, with notes, gzipped output for a .gz name, and a report you can save from the dashboard. Output Files
• Baselines and finer timing: --baseline-file compares a run with an earlier JSON report, times are kept in microseconds, and whole run figures that an audit found wrong are fixed. Metrics
• New command line: Goose parses options with clap, with a short -h and a long --help that explains each option's default and interactions, and --hatch-rate and --startup-time are now --increase-rate and --increase-time, with new --decrease-rate and --decrease-time. Common Options
• More to test: PDF reports, GraphQL requests, and load tests of TCP and other protocols that are not HTTP. Examples

Behavior and API changes

Every change a 0.18 load test may meet. The Book's Upgrading From 0.18 page walks through the ones most load tests need.

Options and the command line

• --hatch-rate (-r) is now --increase-rate and --startup-time (-s) is now --increase-time; GooseDefault::HatchRate and StartupTime are now IncreaseRate and IncreaseTime, the GooseConfiguration fields are renamed to match, and util::get_hatch_rate() is now util::parse_rate(). #687
• --iterations is refused with --decrease-rate or --decrease-time. #781
• --no-autostart is refused when no Controller could bind its port and dashboard control is off. #778
• A time span takes only h, m and s, in either case; values 0.18 misread, such as 15mins or 2h3min, are refused, and util::parse_timespan returns None for them. #778
• Test plan steps take h, m and s in either case. #779
• --pdf-print-html counts as a report for --notes and --no-granular-report, and is refused with --no-metrics. #778
• --scenarios accepts _, so name_1 selects a second scenario registered as name. #778
• --throttle-requests 0 no longer refuses rps thresholds. #778
• GooseConfiguration is parsed by clap: parse_args_default and parse_args_default_or_exit are inherent methods, so drop use gumdrop::Options;, and parse(), try_parse() and try_parse_from() work too. #760
• Many GooseConfiguration fields, among them host, run_time, iterations, the log paths and the Controller hosts and ports, are now Option, None when not given. #760
• The help field is gone; parse_args_default returns GooseError::InvalidOption for -h and --help, and its errors never echo the rest of the command line. #760
• pdf_scale and pdf_timeout are Option<f64> and Option<u64>, and GooseDefault::PdfScale sets the scale in code. #760
• Help is split into a short -h and a long --help, regrouped by task, printed to stdout and wrapped to the terminal. #760
• An option value that begins with a hyphen goes after =, as in --goose-log=-run.log; a negative number is still read as a value. #760
• An output flag followed by another option is read as given without a path, so --goose-log --quiet writes goose.log into the run's folder. #768
• --dashboard-port 0 binds a free port and reports it. #760
• --increase-rate, --decrease-rate and their GooseDefaults are refused below one user a day. #752
• A test plan with no users or too many to count, and a time span or step too large, is an error instead of a panic, including from set_default(GooseDefault::TestPlan, ..). #752
• --scenarios matches names exactly instead of as substrings, and * is a wildcard. #655
• --accept-invalid-certs works with every TLS backend; it was ignored unless the rustls-tls feature was on. The default TLS backend is now rustls. #669

Metrics and reports

• With --co-mitigation, the Aggregated # reqs is actual plus synthetic requests. #778
• Coordinated omission counts, and the errors and transaction errors tables, start again at the metrics reset after ramp up. #778, #779
• A run started by a Controller after another counts its requests from zero. #778
• The reports' Aggregated scenario Iterations matches the console: every run divided by every user. #778
• A scenario counts every user that ran it, not only those that completed an iteration. #778
• Averages with no measured times are 0, not NaN (null in JSON). #778
• Report averages are exact, so the JSON report carries them with more digits. #744
• The JSON report keeps its timing data in microseconds in times_us, with minimum_time_us, maximum_time_us and total_time_us, in place of the millisecond times; minimums, maximums and percentiles stay whole milliseconds, and a 0.18.1 report still loads as a baseline. #767
• The telnet metrics-json and WebSocket metrics replies carry the same microsecond timing data. #767
• GooseRequestMetricTimingData's times, minimum_time, maximum_time and total_time are now _us fields in microseconds, with minimum_time_ms(), maximum_time_ms(), average_ms() and percentile_us() to read them. #767
• TransactionMetricAggregate and ScenarioMetricAggregate keep their times in run_times. #767
• StatusCodeTimingSummary and StatusCodeBreakdown times are _us fields in microseconds. #767
• GooseRequestMetric gains response_time_fraction_us, and TransactionMetric and ScenarioMetric gain run_time_fraction_us, so struct literals must set them. #767
• GooseMetric::All holds its scenario in a Box. #767
• GooseMetrics gains transaction_errors, and GooseMetric and GooseLog gain transaction error variants, so an exhaustive match needs new arms. #776
• With the heatmap on, a run that writes a report keeps per second response time histograms, and a user's metrics batch may be sent early. #763
• report.json and the Controller's metrics-json include the test plan history. #768
• A report or log named .gz is written gzipped. #768
• --report-file reports are written even with --no-print-metrics, and --no-granular-report works with --dashboard. #768
• Metrics printed when a --no-autostart run returns to Idle respect --no-print-metrics. #768
• The HTML report's active users graph shows the increase ramp after the default metrics reset. #690
• The metrics::delta and metrics::nullable modules are private, so DeltaValue, Value, ApplyBaseline and NullableFloat are no longer public. #660

Thresholds and the killswitch

• execute() returns GooseError::ThresholdsFailed after the reports are written when a threshold fails, so a main using ? exits nonzero. #773
• GooseMetrics gains thresholds, and the new goose::threshold module holds Threshold and ThresholdResult. #773
• GooseConfiguration gains threshold, no_threshold_abort and threshold_abort_delay, and GooseDefault the matching variants. #773
• The killswitch keeps its first reason and is cleared when a run starts, and reset_killswitch() is public. #768

Output folders

• GooseConfiguration gains output_dir, compress, notes, report_graph_data and render_report, and GooseDefault the matching variants. #768
• config::RUN_FOLDER is public and marks an output flag given without a path. #768
• An autostarted --dashboard run that ends on its own with a client connected stays up until Quit, Ctrl-C or shutdown. #768

Controllers

• The WebSocket Controller is off by default; --websocket turns it on. #782
• The telnet and WebSocket Controllers listen on 127.0.0.1 by default instead of 0.0.0.0, and warn when listening off loopback. #782
• A set dashboard auth token shows as (set) in a Controller's config and in {:?} of GooseConfiguration or GooseUser. #782
• users refuses 0 and anything above 100000. #729
• users on a running ramp keeps its run time and ramp down, and is refused while a test plan runs; a rate or time is refused while a test plan is configured. #761
• runtime on a running ramp changes the current run; runtime 0 holds until stopped. #762
• A decrease rate on a running ramp takes effect at once. #761
• The command line's rules between options also apply to Controller commands, and out of range hosts, rates, times and test plans are refused. #761
• increaserate and decreaserate must be at least one user a day; 0 and absurd values are refused. #752
• test-plan must start 1 to 100000 users and is refused while stopping and in Shutdown. #752
• Refusals say what the value must be or why a Start was refused, instead of generic replies. #752
• A Start with no users to start is refused instead of hanging. #752
• Rates and times are stored as parsed, their units are not case sensitive, and a zero such as 0s means unset. #752
• A stop, shutdown or Ctrl-C with no user running ends the load test. #752
• A WebSocket message over 131,072 bytes closes the connection with code 1009. #752
• The config JSON has no help key, lists keys in help order, and shows unset options as null. #760
• ControllerCommand carries its parsed value, adds Quit, drops Help and Exit, and derives only Clone and Debug; TestPlan::total_users saturates, and GooseError::InvalidControllerCommand is no longer returned. #752
• The --no-autostart error without a Controller or dashboard control has new wording. #690
• The dashboard's Target and step fields are read only, with the reason shown, when they cannot be changed. #761
• The dashboard no longer serves /static/chart.min.js; Chart.js is bundled in app.js. #732

Transactions, users and logs

• Transaction::set_name() names only the transaction, not its requests; use set_name_for_transaction_and_requests() for the old behavior. #580
• Transaction names are a TransactionName, GooseRequestMetric.transaction_index is a usize, TransactionDetail has no lifetime, and request and scenario names are Arc<str>. #580
• Scenario, ScenarioMetric, TransactionMetric and their aggregates hold names as Arc<str> instead of String. #683
• Session data types must implement Clone. #656
• TransactionError::RequestFailed and LoggerFailed box their payloads. #691
• GooseUser.started is one clock for every user, so log elapsed counts from the start of the load test. #690
• GooseRawRequest.headers is empty unless the request, debug or error log is on. #692
• Keys in a JSON log line follow the message's field order instead of being sorted. #772

PDF reports

• PDF reports use CHROME, then an installed Chrome, Chromium or Edge, then a headless shell Goose downloads once. #755
• A CHROME that does not exist or is not executable is an error, and a macOS app bundle resolves to its binary. #755
• With a .pdf report file, Goose checks that Chrome can render before any user starts. #755

Full changelog

• CHANGELOG.md at 0.19.0
• Compare 0.18.1 to 0.19.0

Contributors

@jeremyandrews, @fabianx-ai, @alecsmrekar, @tstenner (first contribution), @ctron, @slashrsm, @aditanase (first contribution)

Links

• crates.io
• docs.rs
• The Goose Book