Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document the kedro run lifecycle and hooks execution order #1718

Closed
Tracked by #3014
noklam opened this issue Jul 21, 2022 · 4 comments · Fixed by #3482
Closed
Tracked by #3014

Document the kedro run lifecycle and hooks execution order #1718

noklam opened this issue Jul 21, 2022 · 4 comments · Fixed by #3482
Assignees
Labels
Component: Documentation 📄 Issue/PR for markdown and API documentation

Comments

@noklam
Copy link
Contributor

noklam commented Jul 21, 2022

Description

Document the kedro run lifecycle with hooks execution order

This is the latest chart we have (~2 months ago, omitted the after_context_created and after_command_run hooks but we should have that in place too)
image

I think this should be included in the hooks documentation page

Bonus:
Add more examples in the Common Use Cases section.
i.e. If you need to use the context, after_context_created is probably the way to go.

Double bonus:
embed links in the diagram to link to the relevant API docs for the hooks. Maybe like this

Context

This PR enables using Mermaid to draw flow charts in our doc, so we can finally have something that is easier to version control and update.

This is an Mermaid example for our deployment charts

flowchart TD
    A{Can your Kedro pipeline run on a single machine?} -- YES --> B[Consult the single-machine deployment guide];
    B --> C{Do you have Docker on your machine?};
    C -- YES --> D[Use a container-based approach];
    C -- NO --> E[Use the CLI or package mode];
    A -- NO --> F[Consult the distributed deployment guide];
    F --> G[What distributed platform are you using?\n\nCheck out the guides for:\n<li>Argo</li><li>Prefect</li><li>Kubeflow Pipelines</li><li>AWS Batch</li><li>Databricks</li><li>Dask</li></ul>]; 
    style G text-align:left
    H["Does (part of) your pipeline integrate with Amazon SageMaker?<br/><br/>Read the SageMaker integration guide"];
flowchart TD
    A{Can your Kedro pipeline run on a single machine?} -- YES --> B[Consult the single-machine deployment guide];
    B --> C{Do you have Docker on your machine?};
    C -- YES --> D[Use a container-based approach];
    C -- NO --> E[Use the CLI or package mode];
    A -- NO --> F[Consult the distributed deployment guide];
    F --> G[What distributed platform are you using?\n\nCheck out the guides for:\n<li>Argo</li><li>Prefect</li><li>Kubeflow Pipelines</li><li>AWS Batch</li><li>Databricks</li><li>Dask</li></ul>]; 
    style G text-align:left
    H["Does (part of) your pipeline integrate with Amazon SageMaker?<br/><br/>Read the SageMaker integration guide"];

-- If you received an error, place it here.
-- Separate them if you have more than one.
@noklam noklam added the Component: Documentation 📄 Issue/PR for markdown and API documentation label Jul 21, 2022
@antonymilne
Copy link
Contributor

This would be great to have. Just to say that it might be better to do this as two diagrams:

  • one that shows the order of hook execution (useful for anyone wanting to use hooks)
  • one that shows the mapping between hooks and entry points (useful for plugin developers)

Or maybe it's better to colour code (if possible in mermaid) or label somehow each hook with its plugin entry point.

Basically don't feel like you just need to reproduce the diagram exactly as it is in the Miro board. Change things to whatever is clearest and most useful in the docs.

@astrojuanlu
Copy link
Member

Potentially useful: how Hooks themselves are run.

Gave it a quick go, just to see if it has potential:

flowchart LR
  PluginManager -.- pm.hook --> HookRelay
  HookRelay -- foo_hook --> HookCaller --> HookSpec
  HookRelay -- foo_hook2 --> HookCaller -- plugin1 --> HookImpl1
  HookCaller -- plugin2 --> HookImpl2

Originally posted in pytest-dev/pluggy#341 (comment)

@stichbury
Copy link
Contributor

It's almost a happy 1 year birthday for this ticket 🍰

Here is a ticket that I think forms the parent of this one: #1940

@noklam
Copy link
Contributor Author

noklam commented Sep 1, 2023

Let's ship #1940 to get this going since this has a "low" priority and that smaller ticket has a "high" one. This one would be an useful document even it's just for ourselves internally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component: Documentation 📄 Issue/PR for markdown and API documentation
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

5 participants