You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@ion-elgreco and myself are facing a problem with passing storage_options or UPath to a ConfigurableResource while working on IOManagers in dagster-polars.
We are trying to support all fsspec filesystems and multiple storage formats, so the IOManagers need to either take a prepared UPath or storage_options dict. Passing storage_options through normal config fields would not work as it would expose sensitive values in the UI. Using a EnvVar would not work as we need to support an arbitrary dict of potentially sensitive values (and the keys are unknown in advance).
Seems like right now it's impossible to pass an arbitrary raw Python object to a ConfigurableResource?
Having this would allow all UPathIOManager-based IOManagers to benefit from having storage_options available. It would also make building IOManagers (and resources) requiring complex Python objects easier.
Of course, I would not expect these Python objects to show up in the config. I just want a way to pass them.
What's the use case?
I've seen
@ion-elgreco and myself are facing a problem with passing
storage_options
orUPath
to aConfigurableResource
while working on IOManagers indagster-polars
.We are trying to support all
fsspec
filesystems and multiple storage formats, so the IOManagers need to either take a preparedUPath
orstorage_options
dict. Passingstorage_options
through normal config fields would not work as it would expose sensitive values in the UI. Using aEnvVar
would not work as we need to support an arbitrary dict of potentially sensitive values (and the keys are unknown in advance).Seems like right now it's impossible to pass an arbitrary raw Python object to a
ConfigurableResource
?Having this would allow all
UPathIOManager
-based IOManagers to benefit from havingstorage_options
available. It would also make building IOManagers (and resources) requiring complex Python objects easier.Of course, I would not expect these Python objects to show up in the config. I just want a way to pass them.
cc @PedramNavid
Ideas of implementation
I would like to be able to write something like this:
Additional information
No response
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.
The text was updated successfully, but these errors were encountered: