You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In order to be able to make flow heavy dataframes across several components, it could be beneficial to allow to set a compression level to the return object artifacts generated by components, either at the component or pipeline level in the similar way as we can already do with the compression parameter of the Dataset.upload() method.
Motivation
Passing those artifacts from component to others is very handy and allow us to avoid having to define temporary datasets, but since they cannot be compressed make some operations very slow especially when one specific node (eg. experiment GPU node) does not have a datacenter-grade network connectivity.
The text was updated successfully, but these errors were encountered:
MightyGoldenJA
changed the title
Set return object artifact compression artifact
Set return object artifact compression level
Jan 17, 2024
Proposal Summary
In order to be able to make flow heavy dataframes across several components, it could be beneficial to allow to set a compression level to the return object artifacts generated by components, either at the component or pipeline level in the similar way as we can already do with the
compression
parameter of theDataset.upload()
method.Motivation
Passing those artifacts from component to others is very handy and allow us to avoid having to define temporary datasets, but since they cannot be compressed make some operations very slow especially when one specific node (eg. experiment GPU node) does not have a datacenter-grade network connectivity.
The text was updated successfully, but these errors were encountered: