rebar_prv_common_test refactor

[talentdeficit]

• remove partial support for ct test_spec until it can be done properly
  and warn if test_spec is present in test opts
• use new compiler functionality to reduce complexity of provider
• reduce command line options available to those that can be
  supported properly
• support atoms for suite names in ct_tests

[tsloughter]

Hm, could we warn on unsupported command line options? Or would that be too much?

[talentdeficit]

that's already taken care of by getopt, i think

there's a warning for test_spec which is the only ct_run option that currently doesn't work. you can put literally anything else in ct_opts in rebar.config and it'll be passed to the call to ct:run/1, even invalid options (in which case you'll get one of the terrible ct error messages)

of the options i dropped from the command line most were difficult to check for validity or convert to the proper format ct expect from a string. most can be reenabled, i just didn't have time to handle them properly this week

[ferd]

That sounds good. Would have to try it on a work project to feel super safe about it.

[tsloughter]

@talentdeficit can you rebase.

@ferd and you going to try it at work tomorrow?

[ferd]

Yeah, I'll give it a try tomorrow. If I don't have news about this by noon, ping me.

[talentdeficit]

rebased. i also had time to add support for most of the easier command line options and to fix the handling of suites specified as atoms (which ct allows, but rebar3 ct would fail on)

[ferd]

Got this stacktrace while running tests:

===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace
===> Uncaught error: function_clause
===> Stack trace to the error location: [{rebar_prv_common_test,
                                                 format_result,
                                                 [killed],
                                                 [{file,
                                                   "/Users/fhebert/code/self/rebar3/_build/default/lib/rebar/src/rebar_prv_common_test.erl"},
                                                  {line,425}]},
                                                {rebar_prv_common_test,
                                                 '-handle_quiet_results/2-lc$^0/1-0-',
                                                 1,
                                                 [{file,
                                                   "/Users/fhebert/code/self/rebar3/_build/default/lib/rebar/src/rebar_prv_common_test.erl"},
                                                  {line,412}]},
                                                {rebar_prv_common_test,
                                                 handle_quiet_results,2,
                                                 [{file,
                                                   "/Users/fhebert/code/self/rebar3/_build/default/lib/rebar/src/rebar_prv_common_test.erl"},
                                                  {line,412}]},
                                                {rebar_prv_common_test,do,2,
                                                 [{file,
                                                   "/Users/fhebert/code/self/rebar3/_build/default/lib/rebar/src/rebar_prv_common_test.erl"},
                                                  {line,59}]},
                                                {rebar_core,do,2,
                                                 [{file,
                                                   "/Users/fhebert/code/self/rebar3/_build/default/lib/rebar/src/rebar_core.erl"},
                                                  {line,125}]},
                                                {rebar3,main,1,
                                                 [{file,
                                                   "/Users/fhebert/code/self/rebar3/_build/default/lib/rebar/src/rebar3.erl"},
                                                  {line,56}]},
                                                {escript,run,2,
                                                 [{file,"escript.erl"},
                                                  {line,752}]},
                                                {escript,start,1,
                                                 [{file,"escript.erl"},
                                                  {line,276}]}]

[talentdeficit]

{release_shell, true}?

killed isn't a documented return from ct:run_test. what result do you get when running with --verbose?

[ferd]

Trying to dig out of the thousands of log lines we got with the test per usual:

ct_util_server got EXIT from <0.732.0>: killed

14:28:33.095 [error] gen_server test_server_io terminated with reason: killed
14:28:33.113 [error] CRASH REPORT Process test_server_io with 2 neighbours exited with reason: killed in gen_server:terminate/7 line 804
14:28:33.115 [error] CRASH REPORT Process ct_event with 0 neighbours exited with reason: killed in gen_event:terminate_server/4 line 302
14:28:33.129 [error] gen_server test_server_ctrl terminated with reason: killed
14:28:33.131 [error] CRASH REPORT Process test_server_ctrl with 0 neighbours exited with reason: killed in gen_server:terminate/7 line 804

I'm thinking this might be related to cth_readable hook interactions though.

[ferd]

I re-ran it this morning on fresher builds after having master unable to find a few tests and the errors stopped happening. It's possible I was getting stuck on some versioning problem between deps compiled, rebar3 compiled, and the app itself. If stuff was swinging wildly across major versions maybe something funny happened.

[ferd]

Oh, +1 if that wasn't clear.