What is a reasonable limit for a number of partitions in a multi-dimension asset?

[Flamefork]

I read on Slack that there is a general recommendation of 10k partitions max. Does this recommendation apply to multi-dimension assets as a dim1×dim2 limit?

I'm trying to model a multi-tenant daily assets: we keep few years (so ~1k days) of data for ~200-and-growing clients. Since the unit of a multi-stage processing is one day of events of a client, it's very natural to have such partitioning, but I'm worried about this 200k+ partition count. I would appreciate any thoughts or recommendations on this matter.

[philipmunster]

Im in the same situation. Any answer to this?

[bosprimigenious]

Yes—the count is the product of the two dimensions. MultiPartitionsDefinition defines the cross-product, so approximately 1,000 daily partitions × 200 clients gives 200,000 partitions per asset.

An update to the older guidance in this thread: the current documentation recommends 100,000 partitions or fewer per asset, including for MultiPartitionsDefinition:

• MultiPartitionsDefinition API
• Partitioning assets

The guide describes this as a recommendation and warns about slower UI load times above that count. It should not be interpreted as a guarantee that 200,000 partitions will fail—or that everything below 100,000 will perform well.

Since your processing unit is specifically one client-day, I would preserve that granularity initially and benchmark partition-page loading and representative backfills on your actual Dagster version and deployment.

If the results are unacceptable, one modeling option is to split clients into stable groups with separate assets, each retaining date × client partitions. That reduces partitions per asset but increases the asset graph and dependency-management work. Merely making the client dimension dynamic does not remove the cross-product.

I have not benchmarked your 200,000-partition workload; the count and recommendation above are based on the current documentation.

[philipmunster]

Thanks a lot for the clear explanation!