Provide a way to print the reports of the nested tests

[ronisbr]

Hi guys!

I have a package with lots of tests. I am using nested test sets to improve organization, something like:

@testset "Test Set" begin
    @testset "Test Sub Set"
    ...
end

However, if nothing fails, then nothing about the subtests is printed. This is not good, because I cannot be sure if every test was executed. Is it possible to provide an option to @testset print the reports of the subsets all the time?

[ronisbr]

I want to discuss something. I could do this with a very simple patch against v0.6 (can't compile v0.7 sysimg for some reason):

https://gist.github.com/ronisbr/2b973adc074e5a176b124a6886aee23b

With this patch, we can pass a Boolean to @testset to select whether or not we want to print test results even on failure. In this case, if we have something like:

using Base.Test

@testset "Test" true begin
    @testset "Test 1" begin
        @test 1 == 1

        @testset "Teste 1.1" begin
            @test 1 == 1
        end
    end
    @testset "Test 2" begin
        @test 1 == 1
    end
    @testset "Test 3" begin
        @test 1 == 1
    end
    @testset "Test 4" begin
        @test 1 == 1
    end
    @testset "Test 5" begin
        @test 1 == 1
    end
    @testset "Test 6" begin
        @test 1 == 1
    end
end

Then the result will be:

iMac-de-Ronan:tmp ronan.arraes$ julia -J sys.dylib test.jl
Test Summary: | Pass  Total
Test          |    7      7
  Test 1      |    2      2
  Test 2      |    1      1
  Test 3      |    1      1
  Test 4      |    1      1
  Test 5      |    1      1
  Test 6      |    1      1

If the user omit the boolean or choose false, then the default behavior is restored.

I need to figure out what to do when the @testset is a for (currently it prints one messe per loop interaction), but I want to discuss the solution first. Is it OK? Is there a better way to do? When a consensus is reached, I can do a PR or something.

[thofma]

With your patch, do you recurse all the way down or does it print only up to depth 1?

[jgoldfar]

Couldn't this be just be handled by using a custom TestSet?

[ronisbr]

@thofma it goes all the way down. If you have a subtest inside Test 1, for example, then it will print also. It can be better, but I just want to discuss if this is the right way to do.

@jgoldfar I do not know. Do you have a suggestion?

[jgoldfar]

If you look to the example in the documentation, you can customize just about anything about the way that your testset is handled; I don't have a strong opinion on whether the default test framework should provide the option you suggest (though it seems reasonable to keep the default test objects as minimal as possible) but AFAIK it's easy enough to define an alternative testset (even in a package) and provide the option you suggest. If I have time later, I may have a try at doing so out of curiosity, since I've only implemented an AbstractTestSet once before

[ronisbr]

Probably I am wrong, but I did not find an easy way to do this using a custom test set. The only thing I though was to rewrite a custom print_test_results and call it at finish of your new method. But I don't think this is reasonable, since the default print_test_results is already good and you only have to bypass this part of the code:

    # Only print results at lower levels if we had failures
    if np + nb != subtotal
        for t in ts.results
            if isa(t, DefaultTestSet)
                print_counts(t, depth + 1, align,
                    pass_width, fail_width, error_width, broken_width, total_width)
            end
        end
end

However, I thought something different. What if we add an option to print_test_result to do this? The default call from default test set would be without this option to keep the behavior, but a custom test set just need to call it with a different input.

[ronisbr]

I was starting to work with this but my last option is now viable. Because print_test_results are defined only for DefaultTestSet. I am thinking to add a verbose option to print everything. Because, right now, the only solution is to create a new test set and rewrite print_test_results, which I really do not think is good.