Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set return object artifact compression level #1188

Open
MightyGoldenJA opened this issue Jan 17, 2024 · 0 comments
Open

Set return object artifact compression level #1188

MightyGoldenJA opened this issue Jan 17, 2024 · 0 comments

Comments

@MightyGoldenJA
Copy link

Proposal Summary

In order to be able to make flow heavy dataframes across several components, it could be beneficial to allow to set a compression level to the return object artifacts generated by components, either at the component or pipeline level in the similar way as we can already do with the compression parameter of the Dataset.upload() method.

Motivation

Passing those artifacts from component to others is very handy and allow us to avoid having to define temporary datasets, but since they cannot be compressed make some operations very slow especially when one specific node (eg. experiment GPU node) does not have a datacenter-grade network connectivity.

@MightyGoldenJA MightyGoldenJA changed the title Set return object artifact compression artifact Set return object artifact compression level Jan 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant