[BUG] - QHub destroy command is too strict relax destroy to best effort

[costrouc]

OS system and architecture in which you are running QHub

Linux

Expected behavior

Currently QHub destroy fails to continue deletion if is fails deleting a later stage. For example Kubernetes resource deletion fails but due to this failure it stops and doesn't attempt to delete the infrastructure. Changing this behavior to make qhub destroy continue on and attempt to delete as much as it can.

Actual behavior

Stop at first error when destroying.

How to Reproduce the problem?

Run the qhub-integration-tests on aws

Command output

No response

Versions and dependencies used.

No response

Compute environment

No response

Integrations

No response

Anything else?

No response

[costrouc]

There are plenty of examples of this:

• https://github.com/Quansight/qhub-integration-test/runs/5309545141?check_suite_focus=true

Since there are 8 stages we attempt to delete 8, 7, 6, 5, 4, 3, 2, 1 in order. If a single stage fails to delete continue and delete the next one. In the case of AWS and all the cloud providers we really only need 4, 2, 1 to succeed. For a local deployment ideally 8-1 should all succeed.

[costrouc]

When we have an extension mechanism in place this should also allow us to more resiliently delete stages.