Common Test "eating" stacktraces

[robertoaloi]

Hi!

We noticed that Common Test does not preserve stacktraces for throws, but it does for errors. This can complicate debugging.
Some occurrences where this happens:

https://github.com/erlang/otp/blob/master/lib/common_test/src/test_server.erl#L1581-L1585
https://github.com/erlang/otp/blob/master/lib/common_test/src/test_server.erl#L1649-L1652
https://github.com/erlang/otp/blob/master/lib/common_test/src/test_server.erl#L1792-L1793

For example, the following happens if I add a error(foo) in the init_per_testcase/2:

% rebar3 ct --suite apps/els_lsp/test/els_rename_SUITE.erl --case rename_module
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling els_core
===> Compiling els_dap
===> Compiling els_lsp
===> Running Common Test suites...
%%% els_rename_SUITE:
%%% els_rename_SUITE ==> rename_module: SKIPPED
%%% els_rename_SUITE ==> {tc_auto_skip,
    {failed,
        {els_rename_SUITE,init_per_testcase,
            {foo,
                [{els_rename_SUITE,init_per_testcase,2,
                     [{file,
                          "/Users/robertoaloi/git/github/erlang-ls/erlang_ls/apps/els_lsp/test/els_rename_SUITE.erl"},
                      {line,61}]},
                 {test_server,do_init_per_testcase,2,
                     [{file,"test_server.erl"},{line,1553}]},
                 {test_server,run_test_case_eval1,6,
                     [{file,"test_server.erl"},{line,1254}]},
                 {test_server,run_test_case_eval,9,
                     [{file,"test_server.erl"},{line,1224}]}]}}}}

While this happens if I use throw(foo):

% rebar3 ct --suite apps/els_lsp/test/els_rename_SUITE.erl --case rename_module
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling els_core
===> Compiling els_dap
===> Compiling els_lsp
===> Running Common Test suites...
%%% els_rename_SUITE:
%%% els_rename_SUITE ==> rename_module: SKIPPED
%%% els_rename_SUITE ==> {tc_auto_skip,{failed,{els_rename_SUITE,init_per_testcase,foo}}}

We were wondering if there's any specific reason behind this behaviour.

[garazdawi]

I don't know why but I think it is because of backwards compatibility. When common_test was built there either was no try/catch or it was very new and the test case code was thus guarded using a catch. The catch for throws do not have the stacktrace but rather return the value of the throw.

Before test_server and common_test were merged into one application, this behaviour needed to remain in place as the interface in between common_test and test_server was used by third-party applications to customize the behaviour of common_test (very similar to what can be done with ct hooks today).

I think that it should be possible to change this now that common_test and test_server have been merged without breaking backwards compatibility.

[robertoaloi]

Thanks for the quick response @garazdawi ! Is this something the OTP team can take care of or is help needed?

[garazdawi]

Is this something the OTP team can take care of or is help needed?

I'll will defer that question to @u3s and @IngelaAndin as I don't work much with ct nowadays.

[IngelaAndin]

common test issues are currently not that highly prioritized, mostly due to other things being even more important, so I guess if you want to see it fairly soon help is wanted.

[MessageMap]

since the response of the thrown is now a stacktrace. Going to have to change test results for badmatch from common_test sections. Just worried about down stream apps outside of erlang project being a breaking change. Since response from thrown is now a stack.

Will probably update the PR in a day or 2. Thanks again for the help.