Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix dataset factory patterns in Experiment Tracking #1588

Merged
merged 12 commits into from
Oct 27, 2023
1 change: 1 addition & 0 deletions RELEASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ Please follow the established format:
## Bug fixes and other changes
- Fix improper display of 'run-command' inside the metadata panel. (#1569)
- Replace semver with packaging. (#1578)
- Fix dataset factory discovery in Kedro Viz Experiment Tracking. (#1588)

# Release 6.6.0

Expand Down
18 changes: 17 additions & 1 deletion package/kedro_viz/data_access/managers.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"""`kedro_viz.data_access.managers` defines data access managers."""

# pylint: disable=too-many-instance-attributes
# pylint: disable=too-many-instance-attributes, broad-exception-caught
import logging
from collections import defaultdict
from typing import Dict, List, Set, Union
Expand Down Expand Up @@ -82,6 +82,22 @@ def add_catalog(self, catalog: DataCatalog):
if self.tracking_datasets.is_tracking_dataset(dataset):
self.tracking_datasets.add_tracking_dataset(dataset_name, dataset)

def resolve_dataset_factory_patterns(
self, catalog: DataCatalog, pipelines: Dict[str, KedroPipeline]
):
"""Resolve dataset factory patterns in data catalog by matching
them against the datasets in the pipelines.
"""
for pipeline in pipelines.values():
# Temporary try/except block so the Kedro develop branch can work with Viz.
try:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might catch other exceptions. You could try if hasattr(pipeline, 'data_set):`

datasets = pipeline.data_sets()
except Exception: # pragma: no cover
datasets = pipeline.datasets()

for dataset_name in datasets:
catalog.exists(dataset_name)

def add_pipelines(self, pipelines: Dict[str, KedroPipeline]):
"""Extract objects from all registered pipelines from a Kedro project
into the relevant repositories.
Expand Down
4 changes: 4 additions & 0 deletions package/kedro_viz/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,10 @@ def populate_data(
session_class = make_db_session_factory(session_store.location)
data_access_manager.set_db_session(session_class)

# resolve the dataset factory patterns
data_access_manager.resolve_dataset_factory_patterns(catalog, pipelines)

# add catalog and relevant tracking datasets
data_access_manager.add_catalog(catalog)

# add dataset stats before adding pipelines
Expand Down