Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent deploys from failing if pod is evicted #293

Merged
merged 2 commits into from
May 23, 2018

Conversation

dturn
Copy link
Contributor

@dturn dturn commented May 21, 2018

Evictions should be recoverable so don't let them cause a failure.

Fixes: #230

@dturn dturn requested a review from KnVerey May 21, 2018 18:07
@@ -155,6 +155,21 @@ def test_deploy_failed_is_true_for_container_cannot_run_error
assert_equal expected_msg, pod.failure_message
end

def test_deploy_failed_is_false_for_evicted_
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: you've got a trailing underscore (maybe you meant to add "pods"?)

@@ -155,6 +155,21 @@ def test_deploy_failed_is_true_for_container_cannot_run_error
assert_equal expected_msg, pod.failure_message
end

def test_deploy_failed_is_false_for_evicted_
container_state = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did this test really fail without your change? This is setting container status info, but eviction data (as your business logic correctly assumes) is at the top level of the pod status, like this:

status:
  message: 'The node was low on resource: nodefs.'
  phase: Failed
  reason: Evicted
  startTime: 2018-05-04T21:17:08Z

@dturn
Copy link
Contributor Author

dturn commented May 23, 2018

Did this test really fail without your change?

It didn't, probably because I guessed at what the status looks like for an evicted pod. I've updated the test based on kubectl --context tier4 get pod web-5fb98b959-qflv6 -n wedge-viewer-production -o json not sure if its realistic enough, but at least the test fails without my change.

@dturn dturn requested a review from karanthukral May 23, 2018 17:16
@dturn dturn changed the title Deploys are failed if pod is evicted Prevent deploys from failing if pod is evicted May 23, 2018
Copy link
Contributor

@karanthukral karanthukral left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dturn dturn merged commit e8f70f0 into master May 23, 2018
@dturn dturn deleted the improve-eviction-handling branch May 23, 2018 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants