Unclear how to recover from failed instance update

[spadgett]

If I update a service instance and the update fails, the instance is updated with a failed status condition. It's unclear how or if I can recover.

For instance, I tried to update a plan for this specific instance. The update failed. When I tried setting the plan back to the previous value editing the instance YAML, nothing happened.

I1027 12:48:23.621373 40 controller_instance.go:594] ServiceInstance "postgres/dh-rhscl-postgresql-apb-ncjjj": Not processing event because status showed that it has failed

apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceInstance
metadata:
  creationTimestamp: '2017-10-27T12:32:18Z'
  finalizers:
    - kubernetes-incubator/service-catalog
  generateName: dh-rhscl-postgresql-apb-
  generation: 2
  name: dh-rhscl-postgresql-apb-ncjjj
  namespace: postgres
  resourceVersion: '270'
  selfLink: >-
    /apis/servicecatalog.k8s.io/v1beta1/namespaces/postgres/serviceinstances/dh-rhscl-postgresql-apb-ncjjj
  uid: dee1eb80-bb12-11e7-bc97-0242ac110003
spec:
  clusterServiceClassExternalName: dh-rhscl-postgresql-apb
  clusterServiceClassRef:
    name: 27793015fe45db2fbc1deb7372cc4036
  clusterServicePlanExternalName: dev
  clusterServicePlanRef:
    name: 9f90a44d8181941768273a684de50de5
  externalID: 39834740-70ba-49ea-928a-e1d3b605bce0
  parameters: {}
  parametersFrom:
    - secretKeyRef:
        key: parameters
        name: dh-rhscl-postgresql-apbf4lmx
  updateRequests: 1
  userInfo:
    extra:
      scopes.authorization.openshift.io:
        - 'user:full'
    groups:
      - 'system:authenticated:oauth'
      - 'system:authenticated'
    uid: ''
    username: sam
status:
  asyncOpInProgress: false
  conditions:
    - lastTransitionTime: '2017-10-27T12:34:50Z'
      message: >-
        ClusterServiceBroker returned a failure for update call; operation will
        not be retried: Error updating ServiceInstance of ClusterServiceClass
        (K8S: "27793015fe45db2fbc1deb7372cc4036" ExternalName:
        "dh-rhscl-postgresql-apb") at ClusterServiceBroker
        "ansible-service-broker": Status: 400; ErrorMessage: <nil>; Description:
        plan update not possible; ResponseError: <nil>
      reason: UpdateInstanceCallFailed
      status: 'False'
      type: Ready
    - lastTransitionTime: '2017-10-27T12:34:50Z'
      message: >-
        Error updating ServiceInstance of ClusterServiceClass (K8S:
        "27793015fe45db2fbc1deb7372cc4036" ExternalName:
        "dh-rhscl-postgresql-apb") at ClusterServiceBroker
        "ansible-service-broker": Status: 400; ErrorMessage: <nil>; Description:
        plan update not possible; ResponseError: <nil>
      reason: ClusterServiceBrokerReturnedFailure
      status: 'True'
      type: Failed
  externalProperties:
    clusterServicePlanExternalName: prod
    parameterChecksum: f511137c0021f5169de49e662f0ec2830219a26e50968a57f2faa280408dfaa7
    parameters:
      postgresql_database: <redacted>
      postgresql_user: <redacted>
      postgresql_version: <redacted>
    userInfo:
      extra:
        scopes.authorization.openshift.io:
          - 'user:full'
      groups:
        - 'system:authenticated:oauth'
        - 'system:authenticated'
      uid: ''
      username: sam
  orphanMitigationInProgress: false
  reconciledGeneration: 2

@pmorie @jwforres

[pmorie]

Related: #1499

[staebler]

/dibs