How can I load the materialized contents of a specific partition of an asset?

[OwenKephart]

I have a use case in which I'd like to load the materialized value for multiple partitions of an asset. How do I do this?

[OwenKephart]

This can be accomplished using load_asset_value.

For example:

@asset(partitions_def=DailyPartitionsDefinition(start_date="2022-01-01")
def my_asset():
    ...

@op(...)
def my_op():
    from my_dagster_project import defs

    dates = ["2022-01-01", "2022-01-02", ...]
    with defs.get_asset_value_loader() as loader:
        for date in dates:
            value = loader.load_asset_value(AssetKey("my_asset"), partition_key=date)
            do_something_with(value)

[dmsfabiano]

Hi @OwenKephart ! is this still a valid approach? using this gives me

dagster._check.CheckError: Failure condition: The instance is not available to load partitions. You may be seeing this error when using dynamic partitions with a version of dagster-webserver or dagster-cloud that is older than 1.1.18.

[OwenKephart]

@dmsfabiano This is still a valid approach here, although based on your error message, it seems like you might be using dynamic partitions, in which case you should do loader.load_asset_value(AssetKey("my_asset"), partition_key=date, instance=context.instance).

This will allow this call to access the database which Dagster uses to store metadata (including dynamic partitions).

[dmsfabiano]

@OwenKephart although I thought this would be my savior. It looks like context is not a parameter for load_asset_value. How can I load the dynamic partition?
https://github.com/dagster-io/dagster/blob/a0656fce3757af1f680dccf1637f79195bb457a8/python_modules/dagster/dagster/_core/storage/asset_value_loader.py#L72C1-L72C1

[OwenKephart]

Ah my mistake here -- you need to supply the context when creating the AssetValueLoader, not when invoking the function:

@op(...)
def my_op(context: OpExecutionContext):
    from my_dagster_project import defs

    dates = ["2022-01-01", "2022-01-02", ...]
    with defs.get_asset_value_loader(instance=context.instance) as loader:
        for date in dates:
            value = loader.load_asset_value(AssetKey("my_asset"), partition_key=date)
            do_something_with(value)

[dmsfabiano]

@OwenKephart Thank you! that did it, I went through the source code and somehow missed that 😪 Do you think it would be a good idea to either make an adjustment in the docstring of get_asset_value_loader, or defs.load_asset_value or maybe the docs? when I got the original error below, I was somehow lost as where to go to load the dynamic partition. Not sure if this is something I can PR/Write?

dagster._check.CheckError: Failure condition: The instance is not available to load partitions. You may be seeing this error when using dynamic partitions with a version of dagster-webserver or dagster-cloud that is older than 1.1.18.

A separate question, but should dagsterbe able to resolve the instance since the op is running within dagster?