refetch the backfill before updating, to avoid clobbering cancels

[prha]

Summary

The parts of the backfill daemon loop that can take a while is to actually create and submit the runs after reading the partitions from the backfill job.

This diff makes sure to re-fetch the backfill job just before writing state back, to minimize the time window in which the backfill could have been canceled before getting clobbered by the checkpoint write.

Helps address #7090

Test Plan

BK

[vercel]

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployments, click below or on the icon next to each commit.

dagster – ./docs/next

🔍 Inspect: https://vercel.com/elementl/dagster/86DkwfV2tmsWJekZbd8FskwNh8sV
✅ Preview: Canceled

[Deployment for e4541e0 canceled]

dagit-storybook – ./js_modules/dagit/packages/ui

🔍 Inspect: https://vercel.com/elementl/dagit-storybook/6qxpZvfx9Ronrq1EfvXRbCPU1DNs
✅ Preview: Canceled

[Deployment for e4541e0 canceled]

[gibsondan]

the other angle here would be to change our schema so that we don't update the status unless we are specifically changing the status as part of the write, right? Otherwise there's still a potential race here if the cancellation comes in between the read and the write (it's much less frequent though). That would probably require us to add more locking or abandon the serialized-namedtuple-in-the-db pattern

[prha]

Yeah, I did consider that... but it did seem like a lot to pull out the status from the body and / or reconciling it. Still on the table to change the API for updating the object though.