The road to introducing rich into Kedro natively

[datajoely]

Description

Is your feature request related to a problem? A clear and concise description of what the problem is: "I'm always frustrated when ..."

rich is an amazing library that superpowers the terminal experience. It is now mature, stable and Lightweight (212.2 kB, with minimal dependencies).

As a Kedro user one spends most of their time interacting with

1. The click CLI
2. The inspecting the logging outputs produced by the run lifecycle. We actually do have a colour logger in Kedro extras that no one ever uses.

I'm a strong believer that the developer experience in these two areas can be greatly improved by adopting rich natievely.

In my own time I've been working on a kedro-rich plugin, I hope to release this once #1309 is merged but it would also make sense to introduce many of these ideas natively over time with developer experience in mind.

The response to this prototype has been positive on both discord and twitter so I'm pretty confident there is sufficient user evidence for us to pursue this.

Checkpoints

1. Closing related issues and PRs:

• Improve logging setup in Kedro #1461 the topics discussed here will be merged soon and will improve the CLI experience
• Add runner class name to pipeline hooks + introduce after_command_run CLI hook spec #1309 the runner_name key will unblock the release of my plug-in as it will be easy to disable the hooks in Parallel mode.

2. We should migrate the default console logging format using the rich.logging.RichHandler

• It's clever enough to drop down to work in dumb terminals as demonstrated by this CI output.
• It unlocks other fancy rendering mid-run such as progress bars, doing so with traditional console logging doesn't render nicely with the progressbar drawing a new line on each iteration.

3. We should improve the standard log lines so they pretty print during a run:

• kedro/kedro/io/data_catalog.py

  Line 339 in c42ae6d

  "Loading data from `%s` (%s)...", name, type(dataset).__name__

• kedro/kedro/io/data_catalog.py

  Line 377 in c42ae6d

  self._logger.info("Saving data to `%s` (%s)...", name, type(dataset).__name__)

• kedro/kedro/pipeline/node.py

  Line 327 in c42ae6d

  self._logger.info("Running node: %s", str(self))

• Rich allows you to format your logging, backticks like we use above don't get auto-colourized but single quotes would, I patch these in my plug-in, but it's a trivial native change.

4. We should adopt the rich exception handler

• It's a one line change, improves the UX/DX and has a really neat mechanism for suppressing framework frames.

• The example here suppresses click, we could do this as well as suppressing kedro itself. It still shows the frame if the exception is within those libraries, but allows the user to skip to important part of the call stack.

  

5. Improving the CLI experience for users outside of a project directory:

• One of the most common traps for newbies is the way that Kedro behaves inside of a project directory versus outside of one.

• This is because Kedro makes a distinction between 'Global' and 'Project' command groups depending on whether it finds itself in a Kedro directory (i.e. we can detect the correct pyrpoject.toml metadata)

• Whilst not really rich specific, I've used rich markup to make my patch for this print nicely:

  

• This is way better than the current experience which gives the quite confusing, albeit semantically correct error "No such command 'run'".

6. We should introduce progress bars to the runner experience:

• Today the only way to track runner progress is via the log output
• The rich.logging.RichHandler allows us to introduce a complimentary set of rich.progress.Progress bars which will render nicely.
• I've managed to get this sort of working via hooks in my plug-in.
• However, I've struggled to get these working with ParallelRunner. Multiprocessing is weird since it's hard to share memory.
• From a plug-in point of view I think it's fine to say that this isn't supported, but if we were doing this natively we could do so directly within the runner or potentially introduce a set of runner hooks that would allow us to manage state better.
• A really cool end state would be to have a mini-progress bar for each ParallelRunner sub-branch 🚀

7. We could leverage the 3rd party rich-click package:

• The rich-click package is a small but actively maintained library which has reached v1.3.x at the time of writing.
• It is a drop-in replacement for click, all you have to do in theory is import rich_click as click
• If you want to use it piecemeal you can just provide the classes as arguments to the command/group decorators like I do here.
• The main benefit of this library is the fact we get much more readable help messages, like so:
  

[merelcht]

Thanks for writing up this issue so clearly @datajoely! I really like the look of rich and agree that it would improve the CLI experience a lot 😃

I remember when you did the demo for point 2 and changed the default handler in logging.yml to use rich.logging.RichHandler, if we are planning on making logging.yml optional (point two on the to-do list), how would that then work with setting the RichHandler? If making logging.yml optional will cause enabling rich to be a step the user needs to take in order to use rich it might defeat the purpose of making this a better user experience..

[datajoely]

@MerelTheisenQB I can't see any downside of us making the rich handler the default independently of whatever we do with logging.yml

[merelcht]

@MerelTheisenQB I can't see any downside of us making the rich handler the default independently of whatever we do with logging.yml

I meant to say is it still possible to make rich the default handler if we don't have logging.yml? I haven't looked much at how logging works in Kedro btw.

[datajoely]

Oh I see what you mean - I guess this is an implementation detail. I was imagining that all library classes are pre-configured to log nicely in the console. The main ones for me are the kedro.runner.*, kedro.io.data_catalog.*, kedro.pipeline.* and kedro.session.*

Today we set the config here in the session:

kedro/kedro/framework/session/session.py

Line 193 in c42ae6d

def _get_logging_config(self) -> Dict[str, Any]:

I was imagining that we would still have a default console logging configuration that could be overridden if they user wanted to, we could even default to the logging template within Kedro already.

[noklam]

#1461

I think Merel is asking this because earlier today we have this coming task is to make logging.yml optional.

[datajoely]

So I guess I'm okay with binning the default logging.yml it's original purpose isn't super valid anymore. I'm advocating making our default console logging handler the rich one.

[antonymilne]

Words cannot express how much I love everything about this. Really amazing work on the plugin and beautiful write-up here! 🚀 😮

One small question: I see that in some places you hard code colours like [orange1 bold]. Is it possible to do this in some way that will render nicely for those poor unfortunates who have a light background terminal? I don't think it's worth making our own custom system that does this by manually doing f"{colour_1}" every time you need a colour, but is there something built into rich that will provide a generic palette of colours that works for different backgrounds?

On the logging.yml point: your suggestion is definitely compatible with the changes we're making. Basically we're removing the project-side logging.yml (though users can still create it if they like) but keeping the framework-side one. So we would just change the framework-side console handler to use rich.logging.RichHandler and make sure there's docs on how people can change that if they want (by making a custom project-side logging.yml).

[datajoely]

So @AntonyMilneQB I have some very good news for you :)

One small question: I see that in some places you hard code colours like [orange1 bold]. Is it possible to do this in some way that will render nicely for those poor unfortunates who have a light background terminal? I don't think it's worth making our own custom system that does this by manually doing f"{colour_1}" every time you need a colour, but is there something built into rich that will provide a generic palette of colours that works for different backgrounds?

1. Rich is smart enough to drop down to greyscale/unformatted mode if it's working within a dumb terminal - see this CI run and look at the kedro catalog list --format table method

2. That shorthand syntax is easy to write, if you want to do it more programmatically you can use the Text.assemble method to build it more explicitly. The docs on this are here.

[datajoely]

Rich terminal detection docs

[antonymilne]

Most of this has now been implemented. The remaining ones are new issues in https://github.com/kedro-org/kedro/milestone/15.