Passing arbitrary Python objects to ConfigurableResource

[danielgafni]

What's the use case?

I've seen

@ion-elgreco and myself are facing a problem with passing storage_options or UPath to a ConfigurableResource while working on IOManagers in dagster-polars.

We are trying to support all fsspec filesystems and multiple storage formats, so the IOManagers need to either take a prepared UPath or storage_options dict. Passing storage_options through normal config fields would not work as it would expose sensitive values in the UI. Using a EnvVar would not work as we need to support an arbitrary dict of potentially sensitive values (and the keys are unknown in advance).

Seems like right now it's impossible to pass an arbitrary raw Python object to a ConfigurableResource?

Having this would allow all UPathIOManager-based IOManagers to benefit from having storage_options available. It would also make building IOManagers (and resources) requiring complex Python objects easier.

Of course, I would not expect these Python objects to show up in the config. I just want a way to pass them.

cc @PedramNavid

Ideas of implementation

I would like to be able to write something like this:

class MyIOManager(ConfigurableIOManager):
    an_object: PythonObject[MyClass]



my_io_manager = MyIOManager(an_object=MyClass())

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[alangenfeld]

Not the prettiest but does the _ prefix / PrivateAttr stuff work https://docs.dagster.io/concepts/resources#resource-lifecycle ?

[danielgafni]

It won't really work, because I want to pass the Python object to the resource at instantiation time

[danielgafni]

Actually it seems like

my_resource = MyResource(storage_options={"secret": EnvVar("SECRET")})

does work. EnvVar works inside a dict! That's pretty cool.

I think this helps as a workaround for us right now.

However, the ability of passing arbitrary Python objects would still be good to have.