Artillery v2.0.1

CLI & Core

• Add time unit support for think steps in scenarios (#2272)
• metrics-by-endpoint plugin is now enabled by default to capture per-URL metrics in HTTP tests. The metrics will be available only in the JSON report and not output to the console, unless the plugin is loaded explicitly by the user. (#2274)
• Configure CPU and memory for Fargate tests with --cpu and --memory flags (#2301)
• Fix issue with $env not resolving correctly in the config section (#2265)
• Fix an issue with --scenario-name not picking up the correct scenario (#2314)
• Add a new fake-data plugin to allow for realistic test data to be generated in your tests (#2318) (docs)
• Fix handling of metrics with non-alphanumeric characters in ensure checks (#2284)
• Improve error handling in the expect plugin (#2298)

OpenTelemetry

• Add experimental tracing support for Playwright tests, with tracing of test.step() calls, page navigation events, and Core Web Vitals metrics (#2259)
• Improve conformance to OTel semantic conventions:
  • Include error messages for spans with HTTP response codes of 400 or higher (#2285)
  • Include HTTP_REQUEST_CONTENT_LENGTH, HTTP_USER_AGENT and HTTP_FLAVOR attributes (#2285)
• Fix issue with virtual user ID attribute (vu.uuid) not always being set correctly
• Fix inconsistencies in error handling (#2298 & #2285)

Playwright

• Add experimental tracing support via OpenTelemetry with publish-metrics plugin (#2259)
• Fix FID Core Web Vital metric not being emitted as expected

ECS/Fargate

• Configure CPU and memory for Fargate tests with --cpu and --memory flags (#2301)

Socket.IO

• Allow validation of any number of arguments in response/acknowledge with new args/on options (#2171)
• More flexible validations with ability to wait for multiple responses (#2139)
• Fix issue with displaying match errors in the report

---

This release brought to you by @bernardobridge @InesNi @josh-coderpad @hassy

Artillery v2.0.0-38

New features & improvements

Engines

• playwright: Update Playwright version to latest (1.39.0) (#2230).
• playwright: Add a convenience test.step() helper, allowing you to break your tests into steps that emit custom timing metrics (#2173) - Docs.

CLI/Core

• New CLI flag --scenario-name, allowing you to specify which scenario to run by its name in the test script (#2184) - Docs.
• Expose mean as a metric (#2163).
• Include scenario object in all engine hooks - you can now access this under context.scenario (#2158) - Docs.

Plugins

publish-metrics - OpenTelemetry

• Traces for HTTP requests are now nested under individual scenarios (#2211).
• Include all request metrics in traces: dns_lookup, tcp_handshake, tls_negotiation, request, first_byte, download (#2211).
• Include VU errors in traces (#2211).

Fixes

Artillery Cloud (join the waitlist)

• Ensure all Artillery console logs are sent correctly to Artillery Cloud (#2212).
• Fix issue with phase information not being sent correctly (#2203, #2204).

Engines

• socketio: Allow nested options (e.g. namespace) in emit without throwing validation error (8b3f035, 2c714fc) - Docs.
• socketio: Allow emit action to be an array of strings (#2167) - Docs.

CLI/Core

• fargate: Fix issue with --packages flag.
• fargate: Include apdex plugin and playwright engine by default in image, preventing additional installation in workers.
• Allow unnamed capture groups to be used in capture (#2127).
• Display meaningful error when engine doesn't match the engine provided in config.engine (#2101).

Plugins

• apdex - Allow frustrated, satisfied and tolerated metrics to be checked with ensure plugin (#2210, #2213).
• apdex - Prevent plugin from crashing when using processor functions (#2195).
• publish-metrics - Set headers as metadata for otel-grpc exporter in OpenTelemetry reporter (82cb40b) - Docs.
• publish-metrics - Create endpoint path correctly so it works with managed instances of Dynatrace (ccf6229).
• publish-metrics: Fix OpenTelemetry reporter debug logging (f6ebc70).

---

This release brought to you by @InesNi, @bernardobridge, @kettanaito and @hassy.

Artillery v2.0.0-37

New features & improvements

• Add support for specifying time units in load phase configs (#2084) - (docs)
• Add OpenTelemetry support (#2073, #2113) - (docs)
• Add color output to ensure check summary (#2074)
• Add support for using application keys Datadog appKey support (#2067)
• Update Playwright engine to use Playwright v1.37.1 (#2105)

Other fixes

• Fix issue with tests not running as expected when filenames shared a common prefix (#2092)
• Improve handling of runtime errors in workers on Fargate
• Fix issue with some console logs not showing when running tests on Fargate
• Routine dependency updates

This release brought to you by @bernardobridge @InesNi @yvesDenis @hassy

Artillery v2.0.0-36

New features

• Dynatrace support - the official publish-metrics plugin can now be configured to send Artillery metrics and events to Dynatrace. Read more in the docs. (140ee92)
• config.http.defaults - default headers and cookies for all requests in a test may now be set via config.http.defaults (938f9c4)

Fixes & improvements

Artillery Cloud (join the waitlist)

• Send load phase start/end timings for visualization (#2022)
• Send CLI exit code to display in test run results (407d225)
• Fix test run status not always getting reported correctly (48a4a91)
• Fix bug which caused test runs on AWS Lambda to not be tagged correctly (919327f)

Core

• Allow variables set with --variables to be used anywhere in config (c08c695)
• Fix for encoding of array values in query strings (06989c6)
• Make sure CLI exit handlers are always run (77f6423)
• Fix bug affecting match groups in regexp captures (e92306e)

Plugins

• publish-metrics - Exit early if required config is missing for publish-metrics (7871d39)
• publish-metrics - Reduce memory usage of the plugin (338b070)
• expect - add useRequestNamesOnly option (#1995)
• ensure - fix bug where ensure config under config.plugins.ensure would not get picked up (d1136f3)

This release brought to you by @InesNi @bernardobridge @hassy

Artillery v2.0.0-35

New features

• Splunk and New Relic integrations now have support for sending test started/finish events via publish-metrics (#1963 & #1958)

Fixes & improvements

• Fix issue with sending reports to Mixpanel in publish-metrics (#1986)
• Fix issue with ensure plugin reading its settings from config.ensure
• Fix issue with sending test metadata to Artillery Cloud (#1954)
• Fix issue with config.payload used along with config.environments (#1964)
• Dependency updates to address security warnings in dependencies

---

With contributions from: @bernardobridge @InesNi @hassy

Skytrace v0.0.7

Fixes & Improvements

• Fix an issue with handling of JSON responses which would make JSON-based expectations and checks fail even when the response was correct JSON

Skytrace v0.0.6

New features

JMESPath expectations

JMESPath is a popular query language for JSON supported by many tools such as AWS CLI and Azure CLI.

Skytrace now supports for setting expectations via JMESPath expressions. An expression that evaluates to a truthy value will result in a successful check.

# Check that the first object in the list of results has a title field:
skytrace http get http://lab.artillery.io/movies -e 'jmespath: "[0].title != null"'

Connected to http://lab.artillery.io

HTTP/1.1 200 OK
x-powered-by: Express
content-type: application/json; charset=utf-8
content-encoding: gzip
etag: W/"558f-PLaaZcOcQVqOfQBMyp4Dw3RncCE"
date: Thu, 22 Jun 2023 14:07:01 GMT
connection: keep-alive
keep-alive: timeout=5
transfer-encoding: chunked
server: Fly/a0b91024 (2023-06-13)
via: 1.1 fly.io
fly-request-id: 01H3HPW44SMJEXCJ6KSVRPQXPM-ams


 DNS Lookup | TCP Connection | SSL Handshake | Time to First Byte | Content Transfer
  1ms       |  35ms          |     N/A       |     53ms           |   12ms          |
            |                |               |                    |                 |
            1ms              |               |                    |                 |
                          36ms               |                    |                 |
                                           36ms                   |                 |
                                                                89ms                |
                                                                               total:101ms


Body stored in: /var/folders/yd/nyh8vtm17w92h5hm4pdhs8j00000gn/T/2023522-24253-1hq5ccb.f6czg.json

Expectations:

*  ok jmespath [0].title != null

Fixes & improvements

• Print base URL only in "Connecting" message
• Fix POST requests not being available

Artillery v2.0.0-34

New run:lambda command

Tests can now be run on AWS Lambda with the new run:lambda command. (Running tests on AWS Lambda with run --platform aws:lambda will continue working as expected too).

The new command makes running tests on AWS Lambda less verbose and more clear, for example:

artillery run:lambda --region eu-west-1 my-script.yml

Rather than:

artillery run --platform aws:lambda --platform-opt region=eu-west-1 my-script.yml

JMESPath expectations

The expect plugin can now evaluate JMESPath expressions as checks. If the expression evaluates to a truthy value, the check will pass. For example:

- get:
    url: "/foo/1234"
    expect:
      jmespath: "title != null"

More flexibility in metrics-by-endpoint reporting

Two new configuration options are now available to help reduce the number of unique metrics produced by by larger & more complex test suites:

• stripQueryString - ignore all query strings on a URL for purposes of reporting its metrics, i.e. /foo?bar=1 and /foo?baz=2 will be reported as the same /foo endpoint
• and ignoreUnnamedRequests - ignore any requests that don't have a name attribute set explicitly

Other fixes & improvements

• A custom role may be set for AWS Lambda workers with --lambda-role-arn flag for run:lambda command

Community shout outs

@cjsaurusrex

Artillery v2.0.0-33

AWS Fargate support

This release adds support for running tests on AWS Fargate. Scale out your load tests horizontally from any of the six supported regions.

artillery run:fargate --count 10 --region us-east-1 my-script.yml

Supported regions:

• us-east-1
• us-west-1
• eu-west-1
• eu-central-1
• ap-south-1
• ap-northeast-1

AWS Fargate is a serverless solution that requires no infrastructure to be managed, and unlike AWS Lambda does not impose a restriction on the duration of your load tests.

Artillery's Playwright integration is fully supported on Fargate, to let you run browser-based load tests at scale with ease.

Playwright support

Playwright support is now built into Artillery. The integration in artillery-engine-playwright is now bundled into the Artillery CLI.

• The version of Playwright used by the integration has been updated to v1.3.4
• Web Vitals tracking has been improved to provide more accurate measurements
• Playwright scenario file may now be loaded via testFunction attribute rather than flowFunction
• flow attribute may now be omitted in Playwright engine scenarios

Other fixes & improvements

• Pass lambdaRoleArn to AWS Lambda engine correctly (#1920)
• Fix issue with returning matched groups from regexp capture (#1922)
• Track & report downloaded data in HTTP tests via http.downloaded_bytes metric
• Environment variables may now be accessed through the special $env variable (the current way of accessing those via $processEnvironment is still supported but will be deprecated in future)
• Dependency updates to address deprecation and security notices

Contributor shout outs

@tancredi @jpilla @BanksySan

Skytrace v0.0.5

New Features

• Skytrace can run existing Artillery scripts now. Use skytrace run on an existing Artillery script to run it as an end-to-end test.