Independent dynamic partitions within a multi-partition definition

[christeefy]

I am looking for a way to specify an asset that is multi-partitioned by a static partition and a dynamic partition. The catch is that the dynamic partitions are independent across static partitions.

For additional context, my use-case involves supporting multiple machine learning models, partitioned by different customer use-cases. These models are trained independently at different schedules, sometimes ad-hoc. I'm interested in having dynamic partitions so I have a historical record of each use-case's model, for downstream assets.

While Dagster has a way to implement a MultiPartitionDefinition with both a StaticPartitionDefinition and DynamicPartitionDefinition:

partition_def = MultiPartitionsDefinition(
    {
        "static": StaticPartitionsDefinition(["foo", "bar", "baz"]), 
        "dynamic": DynamicPartitionsDefinition(name="dyn")
    }
)

However, the implementation above assumes that all static partitions have the same
dynamic partitions (Image 1). Instead, what I'm interested is to have static partitions that each have their own independent dynamic partitions (Image 2).

Image 1: Multipartition consisting of static and dynamic partitions cartesian products, as provided by MultiPartitionDefinition

Image 2: Static partitions having independent dynamic partitions

---

I'm wondering if the above implementation exists within Dagster's current *PartitionDefintion classes? If not, are there any guidance on how I can implement it?

I'm happy to contribute back to the repo too!

[clairelin135]

Hi @christeefy, sadly this behavior isn't currently supported by the partitioning system.

Appreciate your willingness to contribute. Because this feature is sizable and introduces new surface area, it definitely requires discussion amongst the team to determine this is something we'd like to support.

For now, I'd recommend documenting your use case on the corresponding GH issue: #14228