metadata schema for IO managers

[sryza]

E.g.

@io_manager(output_definition_metadata_schema={"database": str, "schema": str})
def my_io_manager():
    ...

@asset(metadata={"database": "abc", "schema: "xyz"})
def my_asset():
    ...

[chasleslr]

+1

I keep running into this scenario where an IO Manager needs to be configured for a given asset but using metadata for this feels like a "hack" because it bypasses all of the great validation available within Dagster for config schemas. For example, I often want to use a common SparkDataFrameIOManager and configure it differently for certain assets (i.e. partitionBy, mode, etc.), but currently it seems like that either needs to be configured at the instance level, or through asset metadata.

[danielgafni]

Another way of doing this is to return a tuple of your actual object and I/O parameters (saved with pickle). Way more ugly but also more powerful as you can work with complex parameters (arbitrary Python objects).