Add `add_ons` metadata to `pyproject.toml` for project creation #3188

noklam · 2023-10-17T14:53:09Z

NOTE: Kedro datasets are moving from kedro.extras.datasets to a separate kedro-datasets package in
kedro-plugins repository. Any changes to the dataset implementations
should be done by opening a pull request in that repository.

Description

Development notes

I added more comments in the changed files. It's a bit tricky to get everything works

Overall there are few changes and refactoring. There are few benefit of using names over number:

Easy to read from telemetry
Flexible to re-order/change the number. This can be seen from the PR as I remove more code than writing new code.

Move the parsing logic back to starter, cookiecutter will store the parsed list i.e ["Linting","Documentation"]

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

Read the contributing guidelines
Signed off each commit with a Developer Certificate of Origin (DCO)
Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added a description of this change in the RELEASE.md file
Added tests to cover my changes
Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

noklam · 2023-10-17T14:54:02Z

It's functional already, the question is whether we want to keep the options [1,2,3] or the name i.e. ["Linting","Documentation"]. I also have some question around the project creation flow since this is my first ticket related to this theme.

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

noklam · 2023-10-20T16:30:23Z

kedro/framework/cli/starters.py

+ADD_ONS_DICT = {
+    "1": "Linting",
+    "2": "Testing",
+    "3": "Custom Logging",
+    "4": "Documentation",
+    "5": "Data structure",
+}


This need to be used multiple place, probably tests as well.

noklam · 2023-10-20T16:30:40Z

kedro/framework/cli/starters.py

@@ -161,6 +166,50 @@ def _starter_spec_to_dict(
    return format_dict


+def _parse_add_ons_input(add_ons_str: str):


move from the hooks - parsing logic should be in cli.py instead of post_hook

noklam · 2023-10-20T16:33:45Z

kedro/templates/project/hooks/utils.py

        docs_path = current_dir / "docs"
        if docs_path.exists():
            shutil.rmtree(str(docs_path))
    else:
        with open(pyproject_file_path, 'a') as file:
            file.write(docs_pyproject_requirements)

-    if "5" not in selected_add_ons_list:  # If Data Structure not selected
+    if "Data Structure" not in selected_add_ons_list:


Is this name finalised already? I find this a bit awkward and doesn't match the terminology in our docs? cc @amandakys

On #3216 she commented it should be "Data Folder"

noklam · 2023-10-20T16:35:12Z

kedro/framework/cli/starters.py

+    add_ons = config.get("add_ons")
+    if add_ons:
+        config["add_ons"] = [
+            ADD_ONS_DICT[add_on] for add_on in _parse_add_ons_input(add_ons)
+        ]
+        config["add_ons"] = str(config["add_ons"])


Update the cookiecutter.add_ons to the actual name instead of nunber.

noklam · 2023-10-20T16:36:41Z

I will mark as ready for review when the tests are fixed.

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

Merge branch 'noklam/ci-add-on' of https://github.com/kedro-org/kedro into noklam/ci-add-on Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

noklam · 2023-10-20T16:47:54Z

kedro/framework/cli/starters.py

+    "2": "Testing",
+    "3": "Custom Logging",
+    "4": "Documentation",
+    "5": "Data structure",


Data structure -> Data Structure

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

SajidAlamQB

Awesome work @noklam, overall, the refactoring looks great and cleaner. I've tested it manualy and its working as expected. 🌟🌟

kedro/framework/cli/starters.py

SajidAlamQB · 2023-10-23T15:43:04Z

kedro/framework/cli/starters.py

@@ -353,11 +402,20 @@ def _make_cookiecutter_args(
    """
    config.setdefault("kedro_version", version)

+    # Map the selected add on lists to readable name
+    add_ons:str = config.get("add_ons")
+    if add_ons:


Will add_ons ever be null?

noklam · 2023-10-23T16:08:39Z

Removed @merelcht from reviewer list since she is sick. Would be still great to review if this isn't merged yet but I have asked @ankatiyar for review because this PR will block a few other PR so it best to get merged soon.

ankatiyar

Tested this manually. Also good work on refactoring :D

merelcht

LGTM 👍

This might have some impact on #3081, so would be good to coordinate with @lrcouto when it comes to merging.

merelcht · 2023-10-24T12:43:01Z

RELEASE.md

@@ -5,6 +5,7 @@
 * Introduced add-ons to the `kedro new` CLI flow.

 ## Bug fixes and other changes
+- Added a new field `add-ons` to `pyproject.toml` when a project is created.


Nit 🙈

Suggested change

- Added a new field `add-ons` to `pyproject.toml` when a project is created.

* Added a new field `add-ons` to `pyproject.toml` when a project is created.

😅 I keep forgetting this since - is easier to type ...

merelcht · 2023-10-24T12:45:14Z

kedro/templates/project/hooks/utils.py

        docs_path = current_dir / "docs"
        if docs_path.exists():
            shutil.rmtree(str(docs_path))
    else:
        with open(pyproject_file_path, 'a') as file:
            file.write(docs_pyproject_requirements)

-    if "5" not in selected_add_ons_list:  # If Data Structure not selected
+    if "Data Structure" not in selected_add_ons_list:


On #3216 she commented it should be "Data Folder"

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

Signed-off-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com>

Signed-off-by: Nok Lam Chan <nok.lam.chan@quantumblack.com>

noklam added 2 commits October 17, 2023 10:48

release note

ba7becf

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

refactor out the add_ons_dict constant

a7066b9

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

noklam linked an issue Oct 17, 2023 that may be closed by this pull request

Add selected add-ons to the created project's metadata #3129

Closed

noklam added 3 commits October 17, 2023 16:33

refactor the parsing logic and move to cli/starters.py from tepmplate

c725e98

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

use the name of the add_ons

a8ba255

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

remove parsing steps

ed50acd

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

noklam marked this pull request as ready for review October 20, 2023 16:27

noklam requested a review from merelcht as a code owner October 20, 2023 16:27

noklam requested review from SajidAlamQB and merelcht and removed request for merelcht October 20, 2023 16:27

noklam commented Oct 20, 2023

View reviewed changes

noklam removed request for SajidAlamQB and merelcht October 20, 2023 16:34

Merge branch 'develop' into noklam/ci-add-on

a7f525c

noklam marked this pull request as draft October 20, 2023 16:34

noklam commented Oct 20, 2023

View reviewed changes

noklam added 2 commits October 20, 2023 16:37

add comments

e07dd4d

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

h

6f10e2a

Merge branch 'noklam/ci-add-on' of https://github.com/kedro-org/kedro into noklam/ci-add-on Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

noklam commented Oct 20, 2023

View reviewed changes

noklam added 2 commits October 20, 2023 16:48

fix typo

85b6a0c

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

fix type hint

d80ea9b

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

noklam requested review from merelcht and SajidAlamQB October 20, 2023 16:58

noklam marked this pull request as ready for review October 20, 2023 16:58

noklam requested a review from ankatiyar October 23, 2023 13:02

noklam mentioned this pull request Oct 23, 2023

[Parent] - Add-ons Project Creation flow #3216

Closed

10 tasks

SajidAlamQB approved these changes Oct 23, 2023

View reviewed changes

noklam removed the request for review from merelcht October 23, 2023 16:07

Merge branch 'develop' into noklam/ci-add-on

21430f2

ankatiyar approved these changes Oct 23, 2023

View reviewed changes

merelcht approved these changes Oct 24, 2023

View reviewed changes

noklam and others added 3 commits October 24, 2023 13:57

fix release note

41cc528

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

adhoc fix

c43ca7b

Signed-off-by: Nok <nok.lam.chan@quantumblack.com>

Merge branch 'develop' into noklam/ci-add-on

1535c39

Signed-off-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com>

This was referenced Oct 24, 2023

Remove deprecated project_version #3084

Closed

Deprecate project_version and add add_ons to ProjectMetadata #3222

Merged

Merge branch 'develop' into noklam/ci-add-on

c7d9da7

Signed-off-by: Nok Lam Chan <nok.lam.chan@quantumblack.com>

noklam merged commit 00f035a into develop Oct 24, 2023
52 of 58 checks passed

noklam deleted the noklam/ci-add-on branch October 24, 2023 19:09

SajidAlamQB mentioned this pull request Oct 25, 2023

Move tests folder to be aligned with develop template kedro-org/kedro-starters#170

Merged

3 tasks

merelcht mentioned this pull request Oct 25, 2023

Add selected add-ons to the created project's metadata #3129

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `add_ons` metadata to `pyproject.toml` for project creation #3188

Add `add_ons` metadata to `pyproject.toml` for project creation #3188

noklam commented Oct 17, 2023 •

edited

Loading

noklam commented Oct 17, 2023

noklam Oct 20, 2023

noklam Oct 20, 2023

noklam Oct 20, 2023

merelcht Oct 24, 2023

noklam Oct 20, 2023

noklam commented Oct 20, 2023

noklam Oct 20, 2023

SajidAlamQB left a comment •

edited

Loading

SajidAlamQB Oct 23, 2023

noklam commented Oct 23, 2023

ankatiyar left a comment

merelcht left a comment

merelcht Oct 24, 2023

noklam Oct 24, 2023

merelcht Oct 24, 2023

		@@ -161,6 +166,50 @@ def _starter_spec_to_dict(
		return format_dict


		def _parse_add_ons_input(add_ons_str: str):

	- Added a new field `add-ons` to `pyproject.toml` when a project is created.
	* Added a new field `add-ons` to `pyproject.toml` when a project is created.

Add add_ons metadata to pyproject.toml for project creation #3188

Add add_ons metadata to pyproject.toml for project creation #3188

Conversation

noklam commented Oct 17, 2023 • edited Loading

Description

Development notes

Developer Certificate of Origin

Checklist

noklam commented Oct 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noklam commented Oct 20, 2023

Choose a reason for hiding this comment

SajidAlamQB left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

noklam commented Oct 23, 2023

ankatiyar left a comment

Choose a reason for hiding this comment

merelcht left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Add `add_ons` metadata to `pyproject.toml` for project creation #3188

Add `add_ons` metadata to `pyproject.toml` for project creation #3188

noklam commented Oct 17, 2023 •

edited

Loading

SajidAlamQB left a comment •

edited

Loading