Skip to content

Commit

Permalink
Fix issue where we were calling the partition function n times, where…
Browse files Browse the repository at this point in the history
… n is the number of partitions returned by the function (#7539)

Summary:
The old logic is not efficient when this function does non-trivial work.

Test Plan: Existing BK, run a schedule that returns multiple partitions and verify with logs that it is no longer called in a loop
  • Loading branch information
gibsondan committed Apr 22, 2022
1 parent d9b523d commit 45db8cd
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion python_modules/dagster/dagster/core/definitions/partition.py
Original file line number Diff line number Diff line change
Expand Up @@ -594,10 +594,12 @@ def _execution_fn(context):
yield SkipReason("Partition selector returned an empty list of partitions.")
return

partition_names = self.get_partition_names(context.scheduled_execution_time)

missing_partition_names = [
partition.name
for partition in selected_partitions
if partition.name not in self.get_partition_names(context.scheduled_execution_time)
if partition.name not in partition_names
]

if missing_partition_names:
Expand Down

0 comments on commit 45db8cd

Please sign in to comment.