@test_logs doesn't work when using local loggers

[veddox]

(See the Discourse discussion here: https://discourse.julialang.org/t/test-logs-behavior-when-using-with-logger/80239)

@test_logs works by creating a TestLogger and wrapping its argument in a with_logger block. The problem with this is that if the argument itself already uses a with_logger block, this inner block receives all log statements without passing them on to the TestLogger. Therefore, in such situations @test_logs will fail, even if the underlying code works.

At minimum this should be mentioned in the documentation. However, it would be good to provide a work-around for this. Two suggestions that came to my mind are:

1. Create a @test_output macro to intercept STDOUT. This would be directly equivalent to @test_warn. The advantage would be that this could also be used to test screen output other than log calls. The disadvantage is that having to revert to intercepting STDOUT hides some of the details of log calls.
2. Alternately, one might be able to dig into Base.CoreLogging to provide a way to override local loggers when using @test_logs. This would be the cleanest solution from a user's perspective (@test_logs does what you expect even when using local loggers), but may be tricky/messy to implement.

[veddox]

@c42f, you originally wrote the @test_logs macro, do you have an opinion on this?

[fredrikekre]

The point of @test_logs is that you obtain the log events, not the rendered output, but since a logger doesn't have to print to stdout/stderr at all monitoring stderr doesn't solve the problem anyway.

(If this is for testing a Julia package I suggest you avoid changing logger to begin with, instead just fire the log events with @info etc and let the user/application filter, modify and route them.)

[veddox]

Hi, thanks for your comment!

The point of @test_logs is that you obtain the log events, not the rendered output

Yes, I realise that, that's why I made the second suggestion.

since a logger doesn't have to print to stdout/stderr at all monitoring stderr doesn't solve the problem anyway

From my perspective, there are two ways to address the problem with local loggers:

1. Solve it, so @test_logs does what it says in all situations, even with active local loggers. I'm not sure how this could be done; my suggestion is to go in via Base.CoreLogging (see point 2 above).
2. Work around it. Tell the user that @test_logs doesn't work with local loggers (the documentation currently doesn't mention this), and provide @test_output as a possible workaround for situations in which the local logger is known to print to stdout.

If this is for testing a Julia package I suggest you avoid changing logger to begin with

No, this is for a modelling application I'm working on, and I have specific reasons why I need to work with local loggers. (Although if you look in the Discourse thread above I did manage to circumvent the problem for my specific use-case.)