RESTest 2.2.0

RESTest 2.2.0 finds more kinds of fault, and tells errors, which fail a run, from warnings, which do not. It is the first release you can run with nothing but Docker or nothing but Java; the command line is the same as in 2.1.0, while plans, dictionaries and what an exit code means change as the lines below say.

What is new

• Six more kinds of fault, on by default, each with its switch: a server error other than 500 (F121, checks.otherServerError), a request broken on purpose that the API accepted (F206, checks.invalidDataAccepted), a request that got no reply in time (F312, checks.noReplyInTime), a 501 to an operation the document declares (F205, checks.notImplemented), and a status code (F920, checks.undeclaredStatusCode) or a content type (F921, checks.undeclaredContentType) the document does not declare.
• Errors and warnings: every fault is one or the other, under a title in RESTest's own words and a line on what to look at. A run ends with 1 for errors only: one whose only faults are warnings ends with 0, where 2.1.0 ended it with 1, unless --set checks.failOnWarnings=true makes warnings fail it too. The new errors can turn a 0 into a 1: a run whose only fault is one operation that never answers now ends with 1.
• A switch for every check, the two 2.1.0 already made included (checks.internalServerError, checks.replyOffSchema): --set checks.replyOffSchema=false turns one off for a run.
• What a run could not check, and why, is counted in its summary and in report.json (notChecked); it never fails a run.
• report.json says more about each fault: its severity, title and advice, its key in version 0.9.0 of the Web Fuzzing Commons catalogue of faults, and totals of errors and warnings. It may carry codes from 900 to 999, RESTest's own, whose keys start with RESTEST_, and, like the stored run, timedOutWaitingFor in a failed outcome.
• A container image and archives: ghcr.io/isa-group/restest:2.2.0, for amd64 and arm64, and restest-2.2.0.zip and .tgz below, with launchers for Unix and Windows.
• JSON Schemas of a dictionary and a plan, in schemas/ and in both, so that whoever writes one — a person or a coding agent — can check it before a run.
• Plans are version 2. A plan written for 2.1 is refused; ADR-0034 translates it line by line.
• A dictionary file is keyed by operation and place only (keyedBy: operationAndParameter): any is an operation's name like any other, fuzzing is not a name a file can take, and description is text or nothing.
• RESTest's own plan divides a run 45, 15, 30 and 10 among accepted requests, series, changed requests and pushing, in place of 45, 10, 20 and 25. To divide it as 2.1.0 did, print the plan with --print-campaign, set sequences, mutation and fuzzing to 10, 20 and 25, and hand it back with --campaign.
• A run of a set size: with schedule.stopAfterRequests on, a run stops after schedule.requestsBeforeStopping requests rather than when its time is up.
• Fixes: nullable kept through allOf, oneOf and anyOf; an empty list sent where the document allows no items; a YAML file that contains itself refused rather than crashing; a --url carrying a user name or password refused rather than printed; a document whose first address is only a path tested there; answers still owed waited for as long as the engine would deliver them.

What a run sends and checks can be tuned or switched off: see settings and switches. What it does not do yet is in known limitations.

Install

With Docker:

docker run --rm ghcr.io/isa-group/restest:2.2.0 version

With Java 21 or later, from one of the archives below:

tar -xzf restest-2.2.0.tgz
restest-2.2.0/bin/restest version

From the source:

git clone --branch v2.2.0 https://github.com/isa-group/RESTest.git
cd RESTest
./mvnw -q install -DskipTests
./restest version

README · Manual · Reference · All changes