Summary
When any assertion fails, pos-module-tests/modules/tests/public/lib/commands/run.liquid renders the report and then sets response_status 500. platformOS replaces the response body on a 5xx, so the report is thrown away and the caller sees a generic server error with no test name, no field name and no line number.
The effect is that a failing test is indistinguishable from broken infrastructure.
Where
include test_formatter, contracts: contracts, total_errors: total_errors, ...
if total_errors > 0
response_status 500
endif
The formatter runs first and produces a correct body, then the status code discards it.
What the caller actually sees
$ pos-cli test run <env> test/<area>/
Running tests on: https://...
Something went wrong on the server.
We have been notified about it.
Nothing distinguishes this from a runtime error, and nothing identifies the failing assertion.
Why it costs real time
One wrong assertion argument in our suite produced exactly the output above. Because the message reads as a platform fault rather than a test failure, the investigation went to the environment, then to the CLI, then to a deployment bisect. Four hypotheses were raised and disproved before anyone suspected the test.
It also inverts the diagnostic value of the suite: the one moment you most need detail, a red test, is the moment all detail is removed. Once a suite gates deploys, an uninformative red state pushes people toward overriding the gate rather than reading it.
Suggested fix
Return 200 with the report body and let the caller decide from success and total_errors, both of which the JSON formatter already emits and pos-cli already reads. That distinguishes a failing suite from a broken one without depending on the status code.
If a non-2xx is wanted for other consumers, 422 preserves the body on platformOS where 500 does not, or the status could be opt-in via a parameter so the default keeps its diagnostics.
Environment
tests module 1.3.4 from partners.platformos.compos-cli 6.2.4
Related: the JSON formatter also discards per-test results, filed separately, and the two compound. With both present, a red suite reports neither which test failed nor how many assertions ran.
Happy to open a PR.
Summary
When any assertion fails,
pos-module-tests/modules/tests/public/lib/commands/run.liquidrenders the report and then setsresponse_status 500. platformOS replaces the response body on a 5xx, so the report is thrown away and the caller sees a generic server error with no test name, no field name and no line number.The effect is that a failing test is indistinguishable from broken infrastructure.
Where
The formatter runs first and produces a correct body, then the status code discards it.
What the caller actually sees
Nothing distinguishes this from a runtime error, and nothing identifies the failing assertion.
Why it costs real time
One wrong assertion argument in our suite produced exactly the output above. Because the message reads as a platform fault rather than a test failure, the investigation went to the environment, then to the CLI, then to a deployment bisect. Four hypotheses were raised and disproved before anyone suspected the test.
It also inverts the diagnostic value of the suite: the one moment you most need detail, a red test, is the moment all detail is removed. Once a suite gates deploys, an uninformative red state pushes people toward overriding the gate rather than reading it.
Suggested fix
Return 200 with the report body and let the caller decide from
successandtotal_errors, both of which the JSON formatter already emits andpos-clialready reads. That distinguishes a failing suite from a broken one without depending on the status code.If a non-2xx is wanted for other consumers, 422 preserves the body on platformOS where 500 does not, or the status could be opt-in via a parameter so the default keeps its diagnostics.
Environment
testsmodule 1.3.4 from partners.platformos.compos-cli6.2.4Related: the JSON formatter also discards per-test results, filed separately, and the two compound. With both present, a red suite reports neither which test failed nor how many assertions ran.
Happy to open a PR.