Resolve pipeline datasets matching patterns defined in the catalog

[pierre-godard]

Description

When a tracking dataset is defined in the pipeline outputs, but not directly in the data catalog and if it matches a data set pattern defined in the data catalog, then this dataset won't be detected by kedro.

This PR fixes this behavior by adding a dataset resolution step after the catalog and pipelines have been loaded.

Development notes

• The demo-project has been updated to use dataset patterns in the tracking layer.

• Use of the "resolve" term as this is similar to the resolve command mentioned here: Add new catalog CLI commands for dataset factories kedro#2603

QA notes

Checklist

• Read the contributing guidelines
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added new entries to the RELEASE.md file
• Added tests to cover my changes

[ankatiyar]

This is probably not needed!

[ankatiyar]

Suggested change

	for data_set_name in sorted_data_set_names:
	try:
	catalog._get_dataset(data_set_name) # pylint: disable=protected-access
	except DatasetNotFoundError:
	continue
	for dataset in all_datasets:
	catalog.exists(dataset)

[ankatiyar]

This internally calls catalog._get_dataset(dataset) and registers it to the catalog so subsequent catalog.list() calls will have the factory datasets too

[ankatiyar]

Suggested change

	# Sort data sets by name, then by namespace to display similar data sets together
	sorted_data_set_names = sorted(
	data_set_names, key=lambda name: ".".join(reversed(name.split(".")))
	)

[ankatiyar]

ALso just flagging to the team that this method is renamed to pipeline.datasets() on the Kedro develop branch

[ravi-kumar-pilla]

Hi @pierre-godard , Thank you for the PR. WIll it be possible for you to discard the files being renamed from the PR and make changes as suggested by @ankatiyar ?

Also we access all our kedro Viz data via the DataAccessManager. Can you please shift the resolve_dataset_factory_patterns method from server.py to package/kedro_viz/data_access/managers.py.

For your reference based on the suggestions -

def resolve_dataset_factory_patterns(
        self, catalog: DataCatalog, pipelines: Dict[str, KedroPipeline]
    ):
        """Resolve dataset factory patterns in data catalog by matching
        them against the datasets in the pipelines.
        """
        for pipeline in pipelines.values():
                if hasattr(pipeline, 'data_sets'):
                     datasets = pipeline.data_sets()
                else:
                     datasets = pipeline.datasets()
                     
            for dataset_name in datasets:
                catalog.exists(dataset_name)

Thank you

[ravi-kumar-pilla]

Closing this in favor of #1588