Kedro incremental starter

[merelcht]

Description

Kedro starters are used to create projects that contain code to run as-is, or to adapt and extend. They provide pre-defined example code and configuration that can be reused. Some of these starters are also used as part of the Kedro tutorials to teach new users about Kedro.

It's often been said that Kedro has a steep learning curve. The idea of the incremental starter is to have a new starter that gradually brings in more and more components to the Kedro project. The final version of the starter would include all advanced features as well as a full setup for linting, testing and other software engineering best practice tools.

Details

• It should be possible for a user to use the starter at different phases that are increasingly more difficult. So essentially you end up "building" a complex Kedro project through a starter. This could look something like kedro new starter=incremental, kedro starter add step_one etc. We'll need tech design to flesh this out more.
• I think a starter would be suitable for this learning journey because it would enable a user to go back and forth between their own code and the starter code and check it's the same in case of errors.

Questions

• What features should each step/level bring? Complex topics should at least include:
  1. Modular pipelines
  2. Experiment Tracking
  3. Linting
  4. Testing

[datajoely]

I'm interested to see if this library can help https://github.com/cruft/cruft

A related point - I've also seen people ask if they can define the structure of kedro pipeline create themselves.

[yetudada]

I'll close this ticket because this is largely handled by #2388