Replies: 3 comments 1 reply
-
Hey @cbini . I can't promise that we'll be able to get to it soon, but I think this is a reasonable approach. I'm curious - do you ever backfill the data all way way back from the beginning? |
Beta Was this translation helpful? Give feedback.
-
Also, if you don't mind, I changed the name of this discussion to be a little more specific. There's an ancient thing in Dagster called "partition sets", which is basically another name for a PartitionedConfig, so the original title was throwing me off. |
Beta Was this translation helpful? Give feedback.
-
Here's a relevant request:
https://dagster.slack.com/archives/C01U954MEER/p1677535394731719 |
Beta Was this translation helpful? Give feedback.
-
I've been grappling with this issue, and I think a new feature might be the solution.
For one of our systems, we'd like to materialize assets hourly using a partition. However, some of this data can go as far back as 2002, which results in an insane number of partitions.
It would be extremely helpful if we could define a sets of partition types that use start and end dates as boundaries. That way you could define large time windows for historical data and smaller time windows for more-recent and future data.
I imagine it would looks something like this:
Beta Was this translation helpful? Give feedback.
All reactions