Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate overriding default MemoryDataSet #8

Closed
marrrcin opened this issue Mar 18, 2022 · 1 comment · Fixed by #63
Closed

Investigate overriding default MemoryDataSet #8

marrrcin opened this issue Mar 18, 2022 · 1 comment · Fixed by #63
Labels
enhancement New feature or request nice to have It would be nice to have but it's not urgent.

Comments

@marrrcin
Copy link
Contributor

marrrcin commented Mar 18, 2022

Right now the plug-in requires all intermediate datasets to be stored in GCS, which forces the pipeline authors to remember that all intermediate datasets need to be explicitly defined in the data catalog.
Consider overriding the create_default_data_set in the ThreadRunner https://github.com/kedro-org/kedro/blob/74fbd752b6bf77c409f016ed73e60a4ea38d6a95/kedro/runner/thread_runner.py#L51 or find other extension point to make the use of intermediate datasets transparent to the users (we could store them in GCS, as we already have paths to bucket and unique run id in the plugin).

@em-pe
Copy link
Member

em-pe commented Mar 22, 2022

We need to clarify the details and make it environment based as this may not be an optimal setup for a local runs, just the pipelines one.

@em-pe em-pe added enhancement New feature or request nice to have It would be nice to have but it's not urgent. labels Mar 24, 2022
marrrcin added a commit that referenced this issue Aug 1, 2022
marrrcin added a commit that referenced this issue Aug 3, 2022
* #8 Auto-dataset via custom runner
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request nice to have It would be nice to have but it's not urgent.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants