Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[eds] Introduce PodCannotStart condition and canary auto-pause handling #68

Merged

Conversation

xornivore
Copy link
Contributor

What does this PR do?

  • Introduces logic to handle certain failure waiting states as cause for auto-pausing canary.
  • Simplifies logic for reported container states in terminated reason (will basically take anything reported from container).
  • e2e tests for PodCannotStart conditions/scenarios (container config error and image pull errors).

Motivation

Current limitations in support for container waiting state which are abnormal and should result in pausing canary.

Additional Notes

Anything else we should know when reviewing?

Describe your test plan

Scenarios similar to tests in e2e.

@xornivore xornivore added enhancement New feature or request component/controller labels Dec 4, 2020
@xornivore xornivore added this to the v0.5 milestone Dec 4, 2020
@xornivore xornivore requested a review from a team December 4, 2020 04:01
Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pull request contains a valid label.

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pull request contains a valid label.

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pull request contains a valid label.

@xornivore xornivore force-pushed the xornivore/eds-canary-handle-waiting-state-failures branch from a72d934 to ae09707 Compare December 4, 2020 04:05
Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pull request contains a valid label.

@codecov-io
Copy link

codecov-io commented Dec 4, 2020

Codecov Report

Merging #68 (635da0e) into master (c125046) will increase coverage by 0.28%.
The diff coverage is 61.61%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #68      +/-   ##
==========================================
+ Coverage   31.39%   31.68%   +0.28%     
==========================================
  Files          37       37              
  Lines        2758     2818      +60     
==========================================
+ Hits          866      893      +27     
- Misses       1809     1839      +30     
- Partials       83       86       +3     
Flag Coverage Δ
unittests 31.68% <61.61%> (+0.28%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
api/v1alpha1/extendeddaemonset_types.go 100.00% <ø> (ø)
api/v1alpha1/extendeddaemonsetreplicaset_types.go 100.00% <ø> (ø)
api/v1alpha1/zz_generated.deepcopy.go 26.02% <0.00%> (-0.37%) ⬇️
api/v1alpha1/zz_generated.openapi.go 0.00% <0.00%> (ø)
pkg/controller/utils/pod/pod.go 21.83% <18.18%> (-6.40%) ⬇️
controllers/extendeddaemonset/controller.go 67.87% <60.00%> (-0.32%) ⬇️
...trollers/extendeddaemonsetreplicaset/controller.go 53.98% <66.66%> (+0.17%) ⬆️
...ers/extendeddaemonsetreplicaset/strategy/canary.go 75.39% <81.35%> (-0.43%) ⬇️
controllers/extendeddaemonset/utils.go 94.59% <100.00%> (-0.28%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c125046...635da0e. Read the comment docs.

- Introduces logic to handle certain failure waiting states as cause for auto-pausing canary.
- Simplifies logic for reported container states in terminated reason (will basically take anything).
- e2e tests for PodCannotStart scenarios (container config error and image pull errors).
@xornivore xornivore force-pushed the xornivore/eds-canary-handle-waiting-state-failures branch from ae09707 to 466aaa9 Compare December 4, 2020 14:50
Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pull request contains a valid label.

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pull request contains a valid label.

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pull request contains a valid label.

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pull request contains a valid label.

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pull request contains a valid label.

- Improve logging
- Colorize e2e test milestone logs
- Add error logs around reconciler scenarios on update, still require more investigation.
@xornivore xornivore force-pushed the xornivore/eds-canary-handle-waiting-state-failures branch from 635da0e to 3405370 Compare December 8, 2020 03:25
Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pull request contains a valid label.

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pull request contains a valid label.

@xornivore xornivore merged commit efa4090 into master Dec 8, 2020
@xornivore xornivore deleted the xornivore/eds-canary-handle-waiting-state-failures branch December 8, 2020 19:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants