Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1504927 - if apbs fail, mark them as failed. #534

Merged
merged 2 commits into from Nov 3, 2017
Merged

Conversation

jmrodri
Copy link
Contributor

@jmrodri jmrodri commented Nov 2, 2017

We were ignoring an error when an apb failed during provisioning and
marking the apb as completed. This resulted in failed apbs looking as
though they provisioned fine.

Which issue this PR fixes (This will close that issue when PR gets merged)

Fixes #506 - apb provisioning sets ServiceInstance as successful when apb fails
Fixes Bug 1504927 - ASB Failed provision marked successful even on pod error
Fixes Bug 1470851 - Image Pull Error is being marked as Success

@openshift-ci-robot openshift-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Nov 2, 2017
@jmrodri
Copy link
Contributor Author

jmrodri commented Nov 2, 2017

TEST failed provision marked as failed

  • Use your OWN dockerhub organization.
  • Push up the fail-apb, I retagged this one:
    https://hub.docker.com/r/djzager/fail-apb/
  • Deploy local broker as normal, I usually built a new image and configured catasb to use my image.
  • Provision fail-apb from the UI.
  • You will it failed in the logs AND failed in the UI. Prior to my fix, it was incorrectly marked as successful.

@jmrodri
Copy link
Contributor Author

jmrodri commented Nov 2, 2017

TEST image pull

  • User your OWN dockerhub organization
  • Push up the fail-apb (see above)
  • Deploy local broker as normal, I usually built a new image and configured catasb to use my image.
  • Look at the UI to see the fail apb in the list. DO NOT PROVISION
  • Now delete the fail-apb from YOUR dockerhub organization
  • NOW attempt to provision the fail-apb from the UI. It will fail.

@jmrodri
Copy link
Contributor Author

jmrodri commented Nov 2, 2017

Provision an apb that will fail

Provision an apb that will fail

Image pull failed provision

Image pull failed provision

@jmrodri
Copy link
Contributor Author

jmrodri commented Nov 3, 2017

Basically what's happening is that we were never looking at the failed text from the exec AND we ignored the error altogether.

Copy link
Member

@djzager djzager left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, nicely done. However, I want to test this but I can't with catasb. If you could provide the catasb ref to use or rebase your changes, that would be great.

if err != nil {
return nil, err
}

if bindOutput == nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉 good call.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tricky...

@@ -61,12 +62,19 @@ func monitorOutput(namespace string, podname string, log *logging.Logger) ([]byt
credsNotAvailable := errors.New("exit status 2")

output, err := runtime.RunCommand("kubectl", "exec", podname, gatherCredentialsCMD, "--namespace="+namespace)

// cannot exec container, pod is done
podFailed := strings.Contains(string(output), "current phase is Failed")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

We were ignoring an error when an apb failed during provisioning and
marking the apb as completed. This resulted in failed apbs looking as
though they provisioned fine.
podCompleted := strings.Contains(string(output), "current phase is Succeeded") ||
strings.Contains(string(output), "cannot exec into a container in a completed pod")

if err == nil {
log.Notice("[%s] Bind credentials found", podname)
return output, nil
} else if podFailed {
// pod has completed but is in failed state
log.Notice("[%s] APB failed", podname)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be logged as an error?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of log.Notice

Copy link
Member

@djzager djzager left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TESTED :shipit:

@jmrodri jmrodri merged commit c5824e3 into master Nov 3, 2017
@jmrodri jmrodri deleted the bz1504927 branch November 6, 2017 15:45
jianzhangbjz pushed a commit to jianzhangbjz/ansible-service-broker that referenced this pull request May 17, 2018
* Bug 1504927 - if apbs fail, mark them as failed.

We were ignoring an error when an apb failed during provisioning and
marking the apb as completed. This resulted in failed apbs looking as
though they provisioned fine.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

apb provisioning sets ServiceInstance as successful when apb fails
6 participants