Wait for all previous objects to be deleted before continuing

[mattrobenolt]

Related to 3735627

As a side effect of this previous commit, it's possible that the current
task now continues along before all of it's expected objects have
actually been removed. This leads to issues when it tries to delete the
Team/Organization which has dependent objects still remaining to be
deleted.

This just adds a check and a try again later to the task until all of
the previous tasks are actually finished.

@getsentry/infrastructure

---

This change is

[mattrobenolt]

I'm pretty confident that these issues would have eventually resolved themselves since the task would have kept retrying naturally until the objects were actually removed, but this is a bit more sane way to handle it.

[dcramer]

Don't use retry as I'm pretty it has side effects if max retries is enabled

[dcramer]

It's unclear why this should be needed and I'm not convinced we should make the change as it's likely masking something else. Deletions should just delete with the assumption that any children are working correctly

[mattrobenolt]

I could explain better IRL probably. Since this is definitely possible to hit after I fixed a previous issue that was masking this.

[mattrobenolt]

Updated to not use retry, but to enqueue a new task.

[mattrobenolt]

It's unclear why this should be needed and I'm not convinced we should make the change as it's likely masking something else. Deletions should just delete with the assumption that any children are working correctly

The assumption here is a bit wrong. So the problem is that say, inside delete_organization, it in turn calls delete_team to delete it's children. But delete_team doesn't always finish synchronously. It's common and very possible that it defers more tasks to continue the deletion attempts. This leaves teams in DELETION_IN_PROGRESS state, so we need to wait until they're all actually gone and all the subtasks have finished.

[tkaemming]

Reviewed 1 of 1 files at r2.
Review status: all files reviewed at latest revision, 1 unresolved discussion.

---

Comments from Reviewable

[dcramer]

Can we just add return values to delete team / etc

---

Review status: all files reviewed at latest revision, all discussions resolved, all commit checks successful.

---

Comments from Reviewable