caplog: Limit level for caplog but not for pytest UI

[torfsen]

This is a feature request for the caplog fixture.

One can use caplog.set_level and caplog.at_level to temporarily set the logging level during a test case. This sets the desired level on the target logger and resets the logger's original level after the test case (or when the at_level context manager exits).

In my experience, one usually uses caplog to assert that certain important (i.e. high level, such as ERROR) messages are correctly logged. Low-level (e.g. DEBUG) messages are usually not asserted, so ignoring them via caplog.set/at_level makes it easier to write the assertions.

However, when debugging failing test cases, these low-level messages are often very useful in figuring out what went wrong.

With the current functionality, one can either have easily written assertions or helpful low-level log messages for failing tests. It would therefore be nice to be able to limit the level only for the messages available via caplog but not for those displayed in the UI for failing test cases. That is, caplog.records, caplog.record_tuples, and caplog.text would only contain messages at and above the level set by caplog.set/at_level, but for failing test cases the UI would still show all messages based on the logger's own level.

If one also wants to keep the current functionality then instead of changing caplog.set/at_level one could introduce a parameter caplog_only that defaults to False:

caplog.set_level(logging.ERROR)  # Sets the level of the root logger, as before

caplog.set_level(logging.ERROR, caplog_only=True)  # Only limits caplog.records, etc.

This would also make this change backwards-compatible.

[RemiCardona]

I've just hit this, and I must say it took me a while to figure out that --log-cli-level actually influences the caplog fixture. I find this really counter-intuitive that an option to the pytest cli can actually break my logging assertions.

As a user, the unittest API is actually very intuitive and easy to use : with self.assertLogs(logger=None, level=None) captures everything by default and allows restricting the capture to a specific logger and/or level.

However caplog's huge benefit over unittest is that it doesn't consume the messages and allows messages to trickle up to the cli logger, which is a huuuge help for debugging.

So if there's anything I can do to help go in that direction, let me know.