Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1502044 - deprovision fixes #494

Merged
merged 4 commits into from
Oct 17, 2017

Conversation

djzager
Copy link
Member

@djzager djzager commented Oct 16, 2017

Describe what this PR does and why we need it:

This PR is meant to address a few errors with respect to service catalog requests to provision/deprovision.

  1. Return accepted on requests to deprovision when performing async operations
  2. Defer sandbox teardown to prevent leaks and allow proper apb failure handling

@openshift-ci-robot openshift-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 16, 2017
@djzager
Copy link
Member Author

djzager commented Oct 16, 2017

Testing:

  1. Setup cluster w/ service-catalog and ansible-service-broker
  2. Deploy mysql-apb

Results

Before fix (broker):

[2017-10-13T18:50:45.22Z] [NOTICE] ============================================================
[2017-10-13T18:50:45.22Z] [NOTICE]                       DEPROVISIONING
[2017-10-13T18:50:45.22Z] [NOTICE] ============================================================
[2017-10-13T18:50:45.22Z] [NOTICE] ServiceInstance.Id: 2fce0fd03e452dc89d96ad14001197f1
[2017-10-13T18:50:45.22Z] [NOTICE] ServiceInstance.Name: dh-rhscl-mysql-apb
[2017-10-13T18:50:45.22Z] [NOTICE] ServiceInstance.Image: docker.io/ansibleplaybookbundle/rhscl-mysql-apb:latest
[2017-10-13T18:50:45.22Z] [NOTICE] ServiceInstance.Description: Software Collections MySQL APB
[2017-10-13T18:50:45.22Z] [NOTICE] ============================================================
[2017-10-13T18:50:45.22Z] [DEBUG] ExecutingApb:
[2017-10-13T18:50:45.22Z] [DEBUG] name:[ dh-rhscl-mysql-apb ]
[2017-10-13T18:50:45.22Z] [DEBUG] image:[ docker.io/ansibleplaybookbundle/rhscl-mysql-apb:latest ]
[2017-10-13T18:50:45.22Z] [DEBUG] action:[ deprovision ]
[2017-10-13T18:50:45.22Z] [DEBUG] pullPolciy:[ IfNotPresent ]
[2017-10-13T18:50:45.22Z] [DEBUG] role:[ edit ]
172.17.0.2 - - [13/Oct/2017:18:50:45 +0000] "DELETE /ansible-service-broker/v2/service_instances/ab6064c9-b437-4660-92d8-42942470055a?accepts_incomplete=true&plan_id=f318937f8bfd021f2137dda6a3c21723&service_id=2fce0fd03e452dc89d96ad14001197f1 HTTP/1.1" 200 58

After fix (broker):

[2017-10-16T14:14:51.999Z] [NOTICE] ============================================================
[2017-10-16T14:14:51.999Z] [NOTICE]                       DEPROVISIONING
[2017-10-16T14:14:51.999Z] [NOTICE] ============================================================
[2017-10-16T14:14:51.999Z] [NOTICE] ServiceInstance.Id: 2fce0fd03e452dc89d96ad14001197f1
[2017-10-16T14:14:52Z] [NOTICE] ServiceInstance.Name: dh-rhscl-mysql-apb
[2017-10-16T14:14:52Z] [NOTICE] ServiceInstance.Image: docker.io/djzager/rhscl-mysql-apb:latest
[2017-10-16T14:14:52Z] [NOTICE] ServiceInstance.Description: Software Collections MySQL APB
[2017-10-16T14:14:52Z] [NOTICE] ============================================================
[2017-10-16T14:14:52Z] [DEBUG] ExecutingApb:
[2017-10-16T14:14:52Z] [DEBUG] name:[ dh-rhscl-mysql-apb ]
[2017-10-16T14:14:52Z] [DEBUG] image:[ docker.io/djzager/rhscl-mysql-apb:latest ]
[2017-10-16T14:14:52Z] [DEBUG] action:[ deprovision ]
[2017-10-16T14:14:52Z] [DEBUG] pullPolciy:[ IfNotPresent ]
[2017-10-16T14:14:52Z] [DEBUG] role:[ edit ]
172.17.0.2 - - [16/Oct/2017:14:14:51 +0000] "DELETE /ansible-service-broker/v2/service_instances/e50afa17-cb17-4722-9fbe-9d6547604f80?accepts_incomplete=true&plan_id=f318937f8bfd021f2137dda6a3c21723&service_id=2fce0fd03e452dc89d96ad14001197f1 HTTP/1.1" 202 58

After fix (service-catalog):

I1016 14:14:57.031175      62 controller_instance.go:1013] ServiceInstance "foo-project/dh-rhscl-mysql-apb-9lfcv": Polling last operation
I1016 14:14:57.049392      62 controller_instance.go:1107] ServiceInstance "foo-project/dh-rhscl-mysql-apb-9lfcv": Poll returned "in progress" : %!q(*string=<nil>)
I1016 14:14:57.049479      62 controller_instance.go:1193] ServiceInstance "foo-project/dh-rhscl-mysql-apb-9lfcv": last operation not completed (still in progress)
I1016 14:14:57.769105      62 leaderelection.go:199] successfully renewed lease kube-service-catalog/service-catalog-controller-manager
I1016 14:14:59.785646      62 leaderelection.go:199] successfully renewed lease kube-service-catalog/service-catalog-controller-manager
I1016 14:15:01.793427      62 leaderelection.go:199] successfully renewed lease kube-service-catalog/service-catalog-controller-manager
I1016 14:15:03.802499      62 leaderelection.go:199] successfully renewed lease kube-service-catalog/service-catalog-controller-manager
I1016 14:15:05.049706      62 controller_instance.go:921] ServiceInstance "foo-project/dh-rhscl-mysql-apb-9lfcv": Processing
I1016 14:15:05.053346      62 controller.go:312] Creating client for ClusterServiceBroker ansible-service-broker, URL: https://asb.ansible-service-broker.svc:1338/ansible-service-broker/
I1016 14:15:05.053564      62 controller_instance.go:1013] ServiceInstance "foo-project/dh-rhscl-mysql-apb-9lfcv": Polling last operation
I1016 14:15:05.085568      62 controller_instance.go:1107] ServiceInstance "foo-project/dh-rhscl-mysql-apb-9lfcv": Poll returned "failed" : %!q(*string=<nil>)
I1016 14:15:05.085667      62 controller_instance.go:1471] ServiceInstance "foo-project/dh-rhscl-mysql-apb-9lfcv": Setting condition "Ready" to Unknown
I1016 14:15:05.085693      62 controller_instance.go:1490] ServiceInstance "foo-project/dh-rhscl-mysql-apb-9lfcv": Found status change, condition "Ready": "False" -> "Unknown"; setting lastTransitionTime to 2017-10-16 14:15:05.085658578 +0000 UTC
I1016 14:15:05.085717      62 controller_instance.go:1471] ServiceInstance "foo-project/dh-rhscl-mysql-apb-9lfcv": Setting condition "Failed" to True
I1016 14:15:05.085732      62 controller_instance.go:1501] ServiceInstance "foo-project/dh-rhscl-mysql-apb-9lfcv": Setting lastTransitionTime, condition "Failed" to 2017-10-16 14:15:05.085714124 +0000 UTC
I1016 14:15:05.085755      62 controller_instance.go:1521] ServiceInstance "foo-project/dh-rhscl-mysql-apb-9lfcv": Updating status
I1016 14:15:05.085771      62 event.go:218] Event(v1.ObjectReference{Kind:"ServiceInstance", Namespace:"foo-project", Name:"dh-rhscl-mysql-apb-9lfcv", UID:"45eea424-b27c-11e7-853a-0242ac110006", APIVersion:"servicecatalog.k8s.io", ResourceVersion:"82", FieldPath:""}): type: 'Warning' reason: 'DeprovisionCallFailed' ServiceInstance "foo-project/dh-rhscl-mysql-apb-9lfcv": Error deprovisioning: ""
I1016 14:15:05.149930      62 controller_instance.go:217] ServiceInstance "foo-project/dh-rhscl-mysql-apb-9lfcv": Clearing catalog finalizer
I1016 14:15:05.149983      62 controller_instance.go:1521] ServiceInstance "foo-project/dh-rhscl-mysql-apb-9lfcv": Updating status
I1016 14:15:05.349890      62 controller_instance.go:70] ServiceInstance "foo-project/dh-rhscl-mysql-apb-9lfcv": Received delete event; no further processing will occur

executionContext, err := ExecuteApb(
"deprovision", clusterConfig, instance.Spec,
instance.Context, instance.Parameters, log,
)
defer sm.DestroyApbSandbox(executionContext, clusterConfig)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we handle the error, if DestroyApbSandbox has an error? We can make sure that the log of the error happens in the Destroy function, but maybe make it return nothing to make it more clear how to use it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless you want to wrap it in another call just for this area.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

defer func(context ExecutionContenxt config ClusterConfig) {
    err := sm.DestoryApbSandbox(execution, config)
    if err != nil {
         log.Errorf("Error occurred destroying sandbox during deprovision: \n%s", err)
    }
}(executionContext, clusterConfig)

Would this suffice? Should we be taking any additional action in the presence of an error here aside from loudly logging?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated to the DestroyApbSandbox to simply log loudly when it fails and return nothing. Let me know if I should go with this method instead.

Copy link
Contributor

@jmrodri jmrodri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VISIACK

executionContext, err := ExecuteApb(
"deprovision", clusterConfig, instance.Spec,
instance.Context, instance.Parameters, log,
)
defer sm.DestroyApbSandbox(executionContext, clusterConfig)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unless you want to wrap it in another call just for this area.

@openshift-ci-robot openshift-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 16, 2017
@djzager
Copy link
Member Author

djzager commented Oct 16, 2017

Since I added changes for bind/unbind, I'm going to manually test to verify those are still working as expected.

Done. ✅

@shawn-hurley shawn-hurley merged commit f0b6e62 into openshift:master Oct 17, 2017
jianzhangbjz pushed a commit to jianzhangbjz/ansible-service-broker that referenced this pull request May 17, 2018
* Bug 1502044 - return 202 on async deprovision

* Defer sandbox teardown to prevent leaks

* Destroy APB Sandbox should not return anything

* Improve apb error logging
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants