Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#8 Auto-dataset via custom runner #63

Merged
merged 6 commits into from
Aug 3, 2022
Merged

#8 Auto-dataset via custom runner #63

merged 6 commits into from
Aug 3, 2022

Conversation

marrrcin
Copy link
Contributor

@marrrcin marrrcin commented Aug 1, 2022

Description

Back port auto-generating dataset from kedro-azureml to Vertex AI Pipelines. Thanks to this change, user is not forced to add all of the intermediate datasets to the catalog.

Resolves #8

PR Checklist

@marrrcin marrrcin marked this pull request as ready for review August 2, 2022 11:46
@marrrcin marrrcin requested a review from em-pe as a code owner August 2, 2022 11:46
@marrrcin marrrcin changed the title #8 Auto-dataset via custom runner - WIP #8 Auto-dataset via custom runner Aug 2, 2022
@marrrcin marrrcin requested a review from szczeles August 2, 2022 12:40
type: pickle.PickleDataSet
filepath: gs://STAGING_BUCKET/${run_id}/06_models/regressor.pickle
versioned: true
layer: models
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😍

Copy link
Contributor

@szczeles szczeles left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing improvement!

@marrrcin marrrcin merged commit 2b7bdd0 into develop Aug 3, 2022
@marrrcin marrrcin deleted the auto-dataset branch August 3, 2022 10:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Investigate overriding default MemoryDataSet
2 participants