Auto-materialization of non-existing partitions #21812
Labels
area: declarative-automation
Related to Declarative Automation, AutomationConditions and Auto Materialization
type: bug
Something isn't working
Dagster version
1.7.3
What's the issue?
I had an asset with a weekly partitions definitions with start date set to 1900-01-01. It had all partitions ranging back to 1900-01-01 was backfilled or marked as backfilled. When i changed it to have daily partitions instead with a start date of 2024-04-20 and enabled auto-materialization(it was enabled previously as well), dagster started requesting materializations of partitions before 2024-04-20. I also had set "all_partitions" to "False" in my cron-schedule but it seems it still tries to materialize all missing partitions.
I was forced to disable the rule "materialize_on_missing" to prevent it from doing that.
Furthermore I also got the following GraphQL-error when accessing the "Automation"-page of the specific asset:
This might be related to this github issue: Auto-materialization daemon re-runs all partitions if time window is extended to the past
What did you expect to happen?
That it would only try to materialize the latest partition, and not try to catchup. Also not materialize partitions that doesn't exist.
How to reproduce?
Deployment type
Dagster Helm chart
Deployment details
No response
Additional information
No response
Message from the maintainers
Impacted by this issue? Give it a 👍! We factor engagement into prioritization.
The text was updated successfully, but these errors were encountered: