[DataCatalog2.0]: Running pipeline with versioned datasets gives an error

[ElenaKhaustova]

Description

Running pipelines with versioned=True gives an error for all runners using feature-1.0.0 branch

The error exists in Kedro 0.19.12 too.

The following error happens when using SequentialRunner and ThreadRunner:

DatasetError: Save path
'/Users/Projects/Testing/kedrocatalog/data/02_intermediate/preprocessed_companies.parquet/2025-05-13T12.22.11.290Z/prepr
ocessed_companies.parquet' for
ParquetDataset(filepath=/Users/Projects/Testing/kedrocatalog/data/02_intermediate/preprocessed_companies.parquet,
load_args={}, protocol=file, save_args={}, version=Version(load=None, save='2025-05-13T12.22.11.290Z')) must not exist if versioning
is enabled.

With ParallelRunner and SharedMemoryDataCatalog the error changes to:

DatasetError: Data for MemoryDataset has not been saved yet.

Context

Found while #4699

Steps to Reproduce

Set versioned=True for datasets in catalog.yml and run pipelines via Python API:

default = pipelines.get("__default__")
tr = ThreadRunner()
tr.run(pipeline=ds, catalog=catalog) 

[ravi-kumar-pilla]

Hi Team,

I am not sure if this is an expected behavior (Kedro 0.19.12) -

1. Normal kedro run works fine as each run has a new session and catalog
2. In notebook/ipython, we create global instances like catalog when we do kedro jupyter notebook and if we do a run here, the versioned datasets take the catalog's save_version and on the first run it works fine but subsequent runs errors out as below -

a. SequentialRunner -

b. ParallelRunner -

c. ThreadRunner -

Is this expected ?
cc: @DimedS @noklam @ankatiyar

[noklam]

Not sure, was this failing with the earlier versions?

[ravi-kumar-pilla]

Not sure, was this failing with the earlier versions?

I checked on 0.19.5 and it is the same

[ravi-kumar-pilla]

Some more info - #4327
Seems like this is how it was everytime

[merelcht]

I just double checked this as well with Kedro 0.19.2 and the same issue is happening. Since that's the case, we can treat this as a lower priority to fix for the new DataCatalog IMO.