Additional Common Test regression tests

[lrascao]

One test that checks for a failed make, another
that detects a failed test run when only one of several
suites fails.

[ghost]

+1, but it seems to fail on anything older than R15.

[lrascao]

In one of the test cases i have a test.spec file with the relative path of the test location:
{suites,"../itest",[test_SUITE, test2_SUITE]}.

This ../itestpath is correct since the erlang doc says (http://erlang.org/doc/apps/common_test/run_test_chapter.html#id77160):

Any relative paths specified in the test specification, will be relative to the directory which contains the 
test specification file, if ct_run -spec TestSpecFile ... or ct:run:test([{spec,TestSpecFile},...]) executes the 
test. The path will be relative to the top level log directory, if ct:run:testspec(TestSpec) executes the test.

however this doesn't seem to apply to versions older than R15 where what counts is the project root path and not the path of the location of the test spec.
This issue could probably be fixed in rebar so the user doesn't have to worry about writing tests specific for different OTP versions. Maybe we can check the otp version we're running and if below R15 make sure the dir exists and change the relative path.
@Tuncer do you agree?

[ghost]

On 24 January 2016 at 16:58, Luis Rascão notifications@github.com wrote:

@Tuncer do you agree?

If it's as trivial as that, yes, the proposed solution sounds reasonable.

[lrascao]

It turns out that the solution was not as trivial as it seemed, it involved rewriting the .spec file since ct_run reads directly from it, this is not acceptable since it's a user file.
I ended up simply disabling the ct_test_fails for OTP versions older than R16.