You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
sryza
changed the title
Let Jobs select the last partition of an asset
Allow specifying PartitionMapping when non-asset jobs depend on partitioned source assets
Apr 28, 2023
I think the main question here is how exactly the PartitionMapping should be specified. I don't think it makes sense to make it an argument to AssetsDefinition.to_source_asset, because source assets themselves don't have a PartitionMapping. Also, what if you wanted to depend on something that's already a SourceAsset, not an AssetsDefinition?
What's the use case?
I want to use the last partition of an asset as an input to a non-partitioned Job.
This can presently be done using an intermediate asset, however, that way a (potentially big) asset has to be materialized a second time.
Ideas of implementation
My rough idea of what this looks like from the users side would be something like this (using partition mappings):
Additional information
Original discussion: #13918
Related to #13357
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.
The text was updated successfully, but these errors were encountered: