[umbrella ticket] Improve Terraform resource deletion mechanism

[chanwit]

This is the umbrella ticket for label: https://github.com/weaveworks/tf-controller/labels/area%2Fresource-deletion

User Story:

As a Terraform Controller user,
I'd like to have a reliable Terraform resource deletion mechanism,
so that managing the deletion process of my resources becomes more efficient.

Fixing these problems will improve controller architecture Issues belong to three sub-groups, sequenced in order of our work plan:

Single resource deletion

• Cache workspace BLOBs in TF Controller in case the source gets deleted before Terraform CR #632
• cannot delete resource with terraform plan failed when Source not found #454
• Can't delete resource terraform when tf runner pod has issue #666

Resources with dependency deletion

• Azure resource group not being deleted despite destroyResourcesOnDeletion set to true #747
• TF deletion hanged when envFrom are also deleted and destroyResourcesOnDeletion is true #720
• Dependency deletions with destroy: true occur in the wrong order #1019

Namespace deletion (most difficult)

• SPIKE handle a scenario when a namespace is deleted #706
• Deleted and re-created namespace cannot be used until tf-controller restart #641
• Cannot destroy namespace with Terraform CR as ServiceAccount is removed before runner created #584

Optional

• Remove tfstate on Terraform delete #456

Acceptance Criteria:

• Single resource deletion: Improve the reliability of Terraform resource deletion process, using [X] as a metric.
• Resource deletion with dependencies: Ensure reliable handling of dependent resources during deletion, using [X] as a metric.
• Namespace deletion: Ensure reliable handling of deletion of the whole namespace, using [X] as a metric.
• Provide informative error messages for resource deletion failures
• Develop comprehensive documentation on the improved deletion mechanism
• Testing is added
• Address all of the issues listed above