/kind bug

The Cluster object stays in failed state after a credentials problem is resolved.

  failureMessage: |
      Failure detected from referenced resource infrastructure.cluster.x-k8s.io/v1beta1, Kind=OpenStackCluster with name "rotate-creds-cnrwq": failed to reconcile external network: failed to get external network: Unable to re-authenticate: Expected HTTP response code [200] when accessing [GET https://url/v2.0/networks/54258498-a513-47da-9369-1a644e4be692], but got 401 instead
      {"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}: Resource not found: [POST https://url/v3/auth/tokens], error message: {"error":{"code":404,"message":"Could not find Application Credential: OLD ID.","title":"Not Found"}}

What steps did you take and what happened:

Delete used appcreds and wait for failure to accour
Create new app creds and update all resources
Verify that all actions are possible again, on mgmt cluster or in cluster

What did you expect to happen:
Cluster state is good again after fix

Anything else you would like to add:
Looks like Creds errors are handelt as a terminal error and not as a transient error.
Seems to be like a problem we had with the Machines too

Code for the machine problem as an idea how to fix: https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/7a19fb6f038[…]a31df228877a446b9b99/controllers/openstackmachine_controller.go

Code in Openstack cluster controller with the problem: https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/7a19fb6f038[…]a31df228877a446b9b99/controllers/openstackcluster_controller.go

Environment:

• Cluster API Provider OpenStack version (Or git rev-parse HEAD if manually built): v010.3
• Cluster-API version: v1.7.3