Capture warnings and messages and print all at end

[hadley]

Display on summary as W.

[gaborcsardi]

👍

[krlmlr]

Perhaps related: Would it be possible to print a dot or something for each evaluated top-level expression (in addition to the dot for each successful expectation)?

[gaborcsardi]

@krlmlr: That would create a lot of dots imo. Why not just put the ones that you want output for in an expectation, e.g. expect_silent?

[krlmlr]

Agreed. There will be more dots, and it could also hurt test performance. I can print a dot myself with expect_null(NULL), too.

[krlmlr]

I think that both warnings and messages should be handled by the reporter, the default handling for messages would be to muffle them. I'm currently struggling with a case where I'd really like to see the messages that occur within a test.

[hadley]

I think that's reasonable, although I think the default should be to handle messages in the same way as warnings (i.e. with an M and a summary at the end)

[hadley]

This is a first stab at it - please let me know if you have suggestions for improvement

[krlmlr]

Thanks. This looks very nice, I like it.

• Are you also going to handle messages the same way?
• I've looked at dplyr results, most warnings come from testthat: "info"/"label" has been deprecated. I wonder if these should be handled specially.
• The warnings that occur outside of test_that() calls are collected separately.
• Do you plan to treat warnings as errors eventually? I understand that this is not an option for this release.

[hadley]

• I think I'll leave messages for now. They seem like more like output where it's reasonable to ignore them by default - otherwise it makes for frustrating testing.
• I'm as concerned as I used to be about capturing warnings etc that happen outside of tests.
• You mean treat as test failures? Maybe that could be an option, but I don't see it as being particularly important.

[krlmlr]

1. I'd appreciate at least an "M" in the SummaryReporter, perhaps verbose output should be "opt-in".
2. I don't understand this -- does that mean you don't care about what happens outside tests?
3. I'd argue that warning=failure should be the default, but I fear that many packages that use testthat are not ready for this.

[hadley]

1. I'm concerned that even that is too much output for most people.
2. Right - if it's not in a test, why should testthat be involved?
3. It's not clear to me that warnings should be failures; regardless, I think it's too late to change now.