You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Right now the plug-in requires all intermediate datasets to be stored in GCS, which forces the pipeline authors to remember that all intermediate datasets need to be explicitly defined in the data catalog.
Consider overriding the create_default_data_set in the ThreadRunnerhttps://github.com/kedro-org/kedro/blob/74fbd752b6bf77c409f016ed73e60a4ea38d6a95/kedro/runner/thread_runner.py#L51 or find other extension point to make the use of intermediate datasets transparent to the users (we could store them in GCS, as we already have paths to bucket and unique run id in the plugin).
The text was updated successfully, but these errors were encountered:
Right now the plug-in requires all intermediate datasets to be stored in GCS, which forces the pipeline authors to remember that all intermediate datasets need to be explicitly defined in the data catalog.
Consider overriding the
create_default_data_set
in theThreadRunner
https://github.com/kedro-org/kedro/blob/74fbd752b6bf77c409f016ed73e60a4ea38d6a95/kedro/runner/thread_runner.py#L51 or find other extension point to make the use of intermediate datasets transparent to the users (we could store them in GCS, as we already have paths to bucket and unique run id in the plugin).The text was updated successfully, but these errors were encountered: