AWS UPI leftovers in claudformations

[petr-balogh]

I've noticed a lot of failed claudformations stacks cause of it failed to delete Hosted Zone.

There are 3 recordsets and if *.apps.jnk-pr2007-b1889.qe.rh-ocs.com. exists, we cannot delete the the hosted zone.

After deleting this 1 recordset, I am able to delete stack

@clacroix12, can you please take a look? And I think we will need to improve the force cleanup script to deal with this.

Thanks

[clacroix12]

@petr-balogh Yep I've definitely noticed this behavior in the past. I'll look into explicitly deleting that recordset before iterating through the cloudformation stacks for deletion.

[petr-balogh]

thanks

[petr-balogh]

Hey @clacroix12 , I've just noticed this one one of the teardown job:

        if http.status_code >= 300:
            error_code = parsed_response.get("Error", {}).get("Code")
            error_class = self.exceptions.from_code(error_code)
>           raise error_class(parsed_response, operation_name)
E           botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjects operation: The specified bucket does not exist

.venv/lib/python3.6/site-packages/botocore/client.py:635: NoSuchBucket
=============================== warnings summary ===============================```

I don't think we should fail the upgrade if bucket not found but just continue here.

So re-opening this.

Job:
https://ocs4-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/qe-destroy-ocs-cluster/7945/console

[clacroix12]

@petr-balogh The error is not related to this issue, but rather an issue with delete_cluster_buckets that was added here

[petr-balogh]

Sorry I didn't get to deep of root cause, just knew about this recent change and thought it's related. So you can close the issue and open the new one please @clacroix12