ct: update auto skip stats when error in suite handled

[hizel]

Hi,

in test case with non-existent group in all() , ct prints Error detected and FAILED, but set exit code set 0 like there was no error here. If ct_run call in ci(like gitlab ci, etc) error can be overlooked.

-module(example_SUITE).
-compile(nowarn_export_all).
-compile(export_all).

all() ->
  [
    {group, exist},
    {group, unexist}
  ].

groups() ->
  [
   {exist, [], [test1]}
  ].

test1(_) ->
  ok.

example ct start

$ ct_run -noshell -dir .  -suite example_SUITE || echo $?


Common Test v1.24 starting (cwd is /home/hizel/tmp/err)


Common Test: Running make in test directories...
Recompile: example_SUITE

CWD set to: "/home/hizel/tmp/err/ct_run.ct@core-hizel.2023-07-10_14.34.38"

TEST INFO: 1 test(s), 0 case(s) in 0 suite(s)

Testing tmp.err.example_SUITE: Starting test, 0 test cases

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Error detected: 'Invalid reference to group unexist in example_SUITE:all/0'
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Testing tmp.err.example_SUITE: *** FAILED {ct_framework,error_in_suite} ***
Testing tmp.err.example_SUITE: TEST COMPLETE, 0 ok, 0 failed of 0 test cases

Updating /home/hizel/tmp/err/index.html ... done
Updating /home/hizel/tmp/err/all_runs.html ... done

[github-actions]

CT Test Results

    2 files    58 suites   1h 26m 52s ⏱️
449 tests 436 ✔️ 13 💤 0 ❌
484 runs  468 ✔️ 16 💤 0 ❌

Results for commit 8f880ec.

♻️ This comment has been updated with latest results.

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

• Complete CT logs (Download Logs)
• HTML Documentation (Download HTML Docs)
• Windows Installer

// Erlang/OTP Github Action Bot

[u3s]

I agree current behavior is not optimal.

Can you fix failing tests? I've spotted problems in:

• ct_auto_compile_SUITE
• ct_test_server_if_1_SUITE

[hizel]

Hi, i rewrite code and add test

[hizel]

Hi, apparently i created pr to the wrong branch master, because this patch not included in the latest release 26.1. How can I fix it?

[u3s]

Hi, apparently i created pr to the wrong branch master, because this patch not included in the latest release 26.1. How can I fix it?

Ahh. I've missed that as well. Please create new PR targeted to maint and we will try to sort it out.