Make kedro micropkg package use pyproject.toml

[astrojuanlu]

Description

Fix gh-2414.

Initially, I tried to be smart and also rework how requirements.txt files are handled. But in the end I opted for this simple solution (just replacing setup.py with pyproject.toml) and it looks like it's simple and it works.

Thanks to gh-2614, not many code changes were required because essentially the pull code is now packaging-agnostic. The only part I had to touch was the on-the-fly generation of the file. Not even the tests needed to be touched.

Development notes

Checklist

• Read the contributing guidelines
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes

[noklam]

I don't have many experience with micropkg before, I have few comments but not necessary related to this PR.

First, I expect kedro pipeline create has symmetry to kedro micropkg package. i.e. kedro pipeline create nok then kedro micropkg package nok should work, but it doesn't because I need to do kedro micropkg pipelines.nok which isn't trivial to find in the doc. I end up have to look at the e2e test. The documentation doesn't tell me I need to do this https://docs.kedro.org/en/latest/nodes_and_pipelines/micro_packaging.html

2. Error message is not very helpful for --all, I am pretty lost when it talks about manifest and

(kedro_core) pattern main % kedro micropkg package
Please specify a micro-package name or add '--all' to package all micro-packages in the 'pyproject.toml' package manifest section.
(kedro_core) pattern main % kedro micropkg package --all
Nothing to package. Please update the 'pyproject.toml' package manifest section.

I also did a manual test to create micropkg and install it

kedro micropkg pipelines.nok
pip install dist/nok-0.1.tar.gz

This works fine but when I try to uninstall it I notice it is also in test folder which I didn't expect.

(kedro_core) pattern main % pip uninstall nok
Found existing installation: nok 0.1
Uninstalling nok-0.1:
Would remove:
/Users/Nok_Lam_Chan/miniconda3/envs/kedro_core/lib/python3.8/site-packages/nok-0.1.dist-info/*
/Users/Nok_Lam_Chan/miniconda3/envs/kedro_core/lib/python3.8/site-packages/nok/*
/Users/Nok_Lam_Chan/miniconda3/envs/kedro_core/lib/python3.8/site-packages/tests/* <- what is this?

[astrojuanlu]

Thanks a lot @noklam !

First, I expect kedro pipeline create has symmetry to kedro micropkg package. i.e. kedro pipeline create nok then kedro micropkg package nok should work, but it doesn't because I need to do kedro micropkg pipelines.nok which isn't trivial to find in the doc.

This is good feedback, we should add that to #1536

Error message is not very helpful for --all, I am pretty lost when it talks about manifest and

This works on main as well right?

I also did a manual test to create micropkg and install it

Couldn't reproduce this 🤔 but I have an idea of how it could have happened.

[merelcht]

Nice and simple solution 👍

I think @noklam raised some very good points, which we need to address. If no tickets exist yet for these issues, let's add them to: https://github.com/kedro-org/kedro/milestone/21

[astrojuanlu]

Spread these comments over gh-1536, gh-2123, gh-2766 👍🏽

[noklam]

Error message is not very helpful for --all, I am pretty lost when it talks about manifest and

This works on main as well right?

I test it with main, the behavior is the same so it is not introduced by this PR, it's just a minor documentation improvement suggestion.

[SajidAlamQB]

Will this still work for users who are still using setup.py in their existing Kedro projects? Do we need to make any considerations for them?

[astrojuanlu]

Yes indeed - in fact, currently the templates still use setup.py (because #2280 isn't closed yet) and this still works.

That's for kedro micropkg package with older projects. For kedro micropkg pull that pulls older micropackages, it will work because after #2614 the machinery is packaging-agnostic. ✨