Fix dataset factory patterns in Experiment Tracking #1588

ravi-kumar-pilla · 2023-10-18T18:14:11Z

Description

Resolves #1480

Development notes

Discovering datasets defined via dataset factory pattern using datasets in pipelines before adding catalog and tracking datasets to kedro viz
Inspired from the PR - Resolve pipeline datasets matching patterns defined in the catalog #1491

Example patterns -

QA notes

Steps to Reproduce
You can also follow the dataset factory patterns guide and modify the demo-project or create a spaceflights project
After executing kedro run, you should see the factory pattern datasets in the experiment tracking run

Checklist

Read the contributing guidelines
Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added new entries to the RELEASE.md file
Added tests to cover my changes

Signed-off-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com>

ankatiyar · 2023-10-19T08:45:36Z

package/kedro_viz/data_access/managers.py

+        """
+        for pipeline in pipelines.values():
+            # Temporary try/except block so the Kedro develop branch can work with Viz.
+            try:


This might catch other exceptions. You could try if hasattr(pipeline, 'data_set):`

noklam · 2023-10-19T12:26:09Z

Is this ready to be reviewed or still in draft?

ravi-kumar-pilla · 2023-10-19T13:40:18Z

Is this ready to be reviewed or still in draft?

I reverted it to draft PR in favor of 1491 . Apologies for the confusion

…x/et-dataset_factory Signed-off-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com>

Signed-off-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com>

…x/et-dataset_factory Signed-off-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com>

Signed-off-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com>

ravi-kumar-pilla · 2023-10-25T20:30:02Z

Is this ready to be reviewed or still in draft?

Hi @noklam, this is now ready for review

ankatiyar

LGTM :)

package/kedro_viz/data_access/managers.py

Signed-off-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com>

…/kedro-viz into fix/et-dataset_factory Signed-off-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com>

noklam · 2023-10-30T13:15:26Z

For reference, @rashidakanchwala shared an alternative solution here. @ravi-kumar-pilla What was the reason that this wasn't accepted, is it because of some edge case?

I guess I don't understand why tracking dataset is treated in a special way, is this just for experiment tracking? Viz should have traversed the graph once already when creating the DAG, so it feels unnecessary to traverse again to resolve the name.

ravi-kumar-pilla · 2023-10-30T14:43:25Z

Hi @noklam , Thanks for mentioning it here. @rashidakanchwala and I had a discussion on this. So, based on what I understood from this PR -

Separation of responsibility - TrackingDatasetRepository was introduced and is populated during the add_catalog phase which is primarily used in experiment tracking runs. (can also be used in pipelines)
Experiment Tracking should not depend on creation of the DAG, rather it is a historical data of kedro runs. Ideally, ET data should not depend on even the catalog information as it is a tracker of historic data and the catalog can change between runs. But, as of now ET does not store data/path to the datasets. This makes it depend on the catalog information.
This PR has the tracking datasets being created during the add_dataset function (i.e., during the creation of node repos for DAG). This makes ET data to be dependant more on the creation of pipelines, which will not serve the purpose of separation of responsibility.

Let me know what do you think of this.

Thank you

noklam · 2023-10-30T16:41:57Z

Experiment Tracking should not depend on creation of the DAG, rather it is a historical data of kedro runs. Ideally, ET data should not depend on even the catalog information as it is a tracker of historic data and the catalog can change between runs. But, as of now ET does not store data/path to the datasets. This makes it depend on the catalog information.

I think this is a good point, thanks for clarifying.

This is minor release with big backend refactoring work and some bug fixes. Bug fixes and other changes Refactor flowchart dataclasses to pydantic base models. (Refactor Flowchart models from dataclass to pydantic base models #1565) Fix dataset factory patterns in Experiment Tracking. (Fix dataset factory patterns in Experiment Tracking #1588) Update demo-project to use OmegaConfigLoader. (Update demo project to use OmegaConfigLoader #1590) Improve feedback for copy to clipboard feature. (Add tooltip to shareable urls copy button #1614) Ensure Kedro-Viz works when hosted on a URL subpath. (Fix: Kedro-Viz doesn't work when hosted via a URL subpath #1621) Bump fastapi upper bounds. (Bump FAST API upper bounds #1634) Fix shareable URL modal to appear across the app. (Make shareable URL modal open globally across the app. #1639) Add Kedro-Viz CLI command deprecation warning. (Add kedro viz deprecation warning for CLI #1641)

ravi-kumar-pilla added 2 commits October 18, 2023 12:39

add dataset factory resolver

87dd132

Signed-off-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com>

fix lint errors

6d1ca97

Signed-off-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com>

ravi-kumar-pilla changed the title ~~Fix issue of dataset factory discovery in Kedro Viz Experiment Tracking~~ Fix for dataset factory discovery in Kedro Viz Experiment Tracking Oct 18, 2023

ravi-kumar-pilla changed the title ~~Fix for dataset factory discovery in Kedro Viz Experiment Tracking~~ Fix dataset factory discovery in Kedro Viz Experiment Tracking Oct 18, 2023

add release notes

2447cc2

Signed-off-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com>

ravi-kumar-pilla marked this pull request as ready for review October 18, 2023 18:17

ravi-kumar-pilla requested review from tynandebold, yetudada and rashidakanchwala as code owners October 18, 2023 18:17

ravi-kumar-pilla requested review from noklam and ankatiyar and removed request for yetudada October 18, 2023 18:18

ravi-kumar-pilla marked this pull request as draft October 18, 2023 19:49

ravi-kumar-pilla mentioned this pull request Oct 18, 2023

Resolve pipeline datasets matching patterns defined in the catalog #1491

Closed

5 tasks

ankatiyar reviewed Oct 19, 2023

View reviewed changes

ravi-kumar-pilla added 5 commits October 23, 2023 15:31

Merge branch 'main' of https://github.com/kedro-org/kedro-viz into fi…

ddfe45c

…x/et-dataset_factory Signed-off-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com>

merge main

4a8fe57

Signed-off-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com>

fix lint

48c1039

Signed-off-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com>

add pytest for resolve dataset factory patterns

9f10bc0

Signed-off-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com>

Merge branch 'main' of https://github.com/kedro-org/kedro-viz into fi…

1bf29d0

…x/et-dataset_factory Signed-off-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com>

ravi-kumar-pilla changed the title ~~Fix dataset factory discovery in Kedro Viz Experiment Tracking~~ Fix dataset factory patterns in Experiment Tracking Oct 25, 2023

modify release statement

7588265

Signed-off-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com>

ravi-kumar-pilla marked this pull request as ready for review October 25, 2023 20:29

tynandebold requested a review from ankatiyar October 26, 2023 11:23

ankatiyar approved these changes Oct 26, 2023

View reviewed changes

Merge branch 'main' into fix/et-dataset_factory

59f218f

noklam reviewed Oct 26, 2023

View reviewed changes

package/kedro_viz/data_access/managers.py Show resolved Hide resolved

noklam reviewed Oct 26, 2023

View reviewed changes

package/kedro_viz/data_access/managers.py Outdated Show resolved Hide resolved

ravi-kumar-pilla added 2 commits October 26, 2023 11:49

add comments based on PR review

ae52534

Signed-off-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com>

Merge branch 'fix/et-dataset_factory' of https://github.com/kedro-org…

3bd3e04

…/kedro-viz into fix/et-dataset_factory Signed-off-by: ravi-kumar-pilla <ravi_kumar_pilla@mckinsey.com>

ravi-kumar-pilla requested a review from noklam October 26, 2023 18:34

noklam approved these changes Oct 27, 2023

View reviewed changes

ravi-kumar-pilla merged commit dca4581 into main Oct 27, 2023
20 checks passed

ravi-kumar-pilla deleted the fix/et-dataset_factory branch October 27, 2023 17:39

rashidakanchwala mentioned this pull request Nov 16, 2023

Release v6.7.0 #1642

Merged

5 tasks

ravi-kumar-pilla mentioned this pull request Jan 9, 2024

Set Kedro Viz start method to spawn process while launching it from Jupyter Notebook #1696

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix dataset factory patterns in Experiment Tracking #1588

Fix dataset factory patterns in Experiment Tracking #1588

ravi-kumar-pilla commented Oct 18, 2023 •

edited

ankatiyar Oct 19, 2023

noklam commented Oct 19, 2023

ravi-kumar-pilla commented Oct 19, 2023

ravi-kumar-pilla commented Oct 25, 2023

ankatiyar left a comment

noklam commented Oct 30, 2023 •

edited

ravi-kumar-pilla commented Oct 30, 2023

noklam commented Oct 30, 2023

Fix dataset factory patterns in Experiment Tracking #1588

Fix dataset factory patterns in Experiment Tracking #1588

Conversation

ravi-kumar-pilla commented Oct 18, 2023 • edited

Description

Development notes

QA notes

Checklist

ankatiyar Oct 19, 2023

Choose a reason for hiding this comment

noklam commented Oct 19, 2023

ravi-kumar-pilla commented Oct 19, 2023

ravi-kumar-pilla commented Oct 25, 2023

ankatiyar left a comment

Choose a reason for hiding this comment

noklam commented Oct 30, 2023 • edited

ravi-kumar-pilla commented Oct 30, 2023

noklam commented Oct 30, 2023

ravi-kumar-pilla commented Oct 18, 2023 •

edited

noklam commented Oct 30, 2023 •

edited