[Draft] - logging

[noklam]

Description

• Consider kedro run -v verbose mode #3591
• How to enable logging automatically #3446

This draft PR is not ready for merge, but rather facilitate the discussion of the issues above.

Development notes

Changes:

1. Read logging.yml automatically - be3d28d
2. Change default of logging.yml- 65cf2a0 . This challenge the idea of "project logging". Why is logger necessary? It's more common to have module level logger, which propagate log to the "root logger", and set the level for root logger only. The exception here is "info logger" because it should only filter INFO level. If anything logger should be optional, not default. I also think it's more likely you want to keep the level consistent for any logger you have manually configured, it's rare you want kedro at INFO but project at DEBUG (This become less important if we end up introduce -v or -vv
3. Add kedro run -v 2ec8bc1 . The main problem for this is CLI argument is slightly too late, there are chance that some log is produced before a "run" start. There is also question of extending this to all CLI command, because logging.yml affect any Kedro CLI. While this introduce a slight inconsistency, I don't think this is a big problem. As I envision this is mainly for debug use.

I imagine -v, -vv is a runtime config that used mostly for debugging. If users has sophisticated configuration, they can continue to use logging.yml to have different group of settings (I haven't seen much tbh)

Alternatlive, I found pip utilise something called a VerboseLogger

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

• Read the contributing guidelines
• Signed off each commit with a Developer Certificate of Origin (DCO)
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes
• Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

[noklam]

Quick notes:
@antonymilne give some good insight:

• Automatic loading logging is fine, but there are edge cases with CONF_SOURCE. I think it's still an improvement and it doesn't make anything worse in the current states.
• We should never change root logger level by default, it's too disruptive. We can however, try to catch kedro-xxx pattern, but always respect the package logging level if it exists.
  • Alternative is we required user to add a new logger in logging.yml manually, would be annoying for plugins.
• kedro run should still default to INFO, so nothing change in terms of default user behavior, -v to change to DEBUG level.

Maybe kedro run -v only affect the project logging, or kedro(and plugins). We should make sure the logs that get produced doesn't print logs from 3rd party package.

[antonymilne]

Some more information now I've found the old issues I wanted to refer to....

• default logging configuration doesn't log other packages than kedro #1732 has a very thorough explanation of why it's a bad idea to set the root logger level to INFO. I don't think it's possible to catch kedro-* as a pattern, but kedro.* is already caught by the kedro logger. This means a plugin called kedro-plugin logging with namespace kedro.plugin which is a bit weird though. Better is to just set the logging level to INFO in kedro-plugin itself, as @Galileo-Galilei did in Set logging level Galileo-Galilei/kedro-mlflow#354
• Work out to show DEBUG level messages emitted early in kedro process #2208 is the only comment I could find that I made on a verbose mode - I thought I had written something else more useful about this but couldn't find it. Basically I think -v/--verbose is a great idea though
• milestones https://github.com/kedro-org/kedro/milestone/5 and https://github.com/kedro-org/kedro/milestone/14 are carefully documented and tracked so definitely worth looking back on the parent issues there if you need more context on how we ended up with the current solution