Skip to content

Commit

Permalink
Add documentation for how plugin CLI commands can be made lazy loaded (
Browse files Browse the repository at this point in the history
…#4027)

* Add docs for making plugins lazy loaded

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Suggestions from reviews:

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

* Small changes

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>

---------

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com>
  • Loading branch information
ankatiyar committed Jul 24, 2024
1 parent e2b20a4 commit 5e15044
Showing 1 changed file with 48 additions and 3 deletions.
51 changes: 48 additions & 3 deletions docs/source/extend_kedro/plugins.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ def commands():
pass


@commands.command()
@commands.command(name="to_json")
@click.pass_obj
def to_json(metadata):
"""Display the pipeline in JSON format"""
Expand All @@ -48,7 +48,6 @@ Commands must be provided as [`click` `Groups`](https://click.palletsprojects.co

The `click Group` will be merged into the main CLI Group. In the process, the options on the group are lost, as is any processing that was done as part of its callback function.


## Project context

When they run, plugins may request information about the current project by creating a session and loading its context:
Expand Down Expand Up @@ -78,7 +77,53 @@ Plugins may also add commands to the Kedro CLI, which supports two types of comm

## Suggested command convention

We use the following command convention: `kedro <plugin-name> <command>`, with `kedro <plugin-name>` acting as a top-level command group. This is our suggested way of structuring your plugin bit it is not necessary for your plugin to work.
We use the following command convention: `kedro <plugin-name> <command>`, with `kedro <plugin-name>` acting as a top-level command group. This is our suggested way of structuring your plugin but it is not necessary for your plugin to work.

## Advanced: Lazy loading of plugin commands

If you are developing a plugin with a large set of CLI commands or with certain large libraries that are slow to import but are used by a small subset of commands, consider using lazy loading of these commands. This can significantly improve the performance of the plugin as well as the overall performance of Kedro CLI. To do this, you can follow [the instructions on the
`click` documentation on lazy loading of commands](https://click.palletsprojects.com/en/8.1.x/complex/#lazily-loading-subcommands). From Kedro 0.19.7, the [Kedro commands are declared as lazy loaded command groups](https://github.com/kedro-org/kedro/blob/main/kedro/framework/cli/cli.py) that you can use as a reference for the implementation.

Consider the previous example of the `kedrojson` plugin. Suppose the plugin has two commands, `kedro to_json pipelines` and `kedro to_json nodes`. The `to_json pipelines` command is used more frequently than the `to_json nodes` command and the `to_json nodes` command requires a large library to be imported. In this case, you can define your commands to be lazily loaded with delayed imports as follows:

In `kedrojson/plugin.py`:
```python
import click
from kedro.framework.project import pipelines
from kedro.framework.cli.utils import LazyGroup

@click.group()
def commands():
pass

@commands.group(
name="to_json",
cls=LazyGroup,
lazy_subcommands={
"nodes": "kedrojson.plugin.nodes",
"pipelines": "kedrojson.plugin.pipelines"
}
)
def to_json():
"""Convert Kedro nodes and pipelines to JSON"""
pass

@click.command(name="nodes")
def nodes():
"""Convert Kedro nodes to JSON"""
import some_large_library
print("Converting nodes to JSON")
...

@click.command("pipelines")
def pipelines():
"""Convert Kedro pipelines to JSON"""
print("Converting pipelines to JSON")
...
```

The loading of the individual `nodes` and `pipelines` commands and subsequently the imports of the large libraries will be delayed until the respective commands are called.


## Hooks

Expand Down

0 comments on commit 5e15044

Please sign in to comment.