Enhancer and HealthUtil now differentiate between fatal and transient errors #1427
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: previously any error returned by the `DefaultEnhancer` was treated as fatal. It made sense because it would act deterministically on a prepared set of resources. However, since #1319 enhancer also uses discovery interface to determine whether or not given resource is namespaced. These API requests may fail and must be retried, hence the introduction of a transient error. Fixes #1413 Signed-off-by: Aleksey Dukhovniy <alex.dukhovniy@googlemail.com>
zen-dog
requested review from
alenkacz,
gerred,
kensipe and
nfnt
as code owners
Removed `engine.ErrTransientExecution` as all non-fatal errors are now treated as transient Signed-off-by: Aleksey Dukhovniy <alex.dukhovniy@googlemail.com>
zen-dog
changed the title
Signed-off-by: Aleksey Dukhovniy <alex.dukhovniy@googlemail.com>
…at are not recoverable Signed-off-by: Aleksey Dukhovniy <alex.dukhovniy@googlemail.com>
zen-dog
commented
Mar 17, 2020
pkg/engine/health/health.go
Outdated
| @@ -83,6 +70,10 @@ func IsHealthy(obj runtime.Object) error { | |||
| log.Printf("HealthUtil: Job \"%v\" is marked healthy", obj.Name) | |||
| return nil | |||
| } | |||
| if terminal, msg := isJobTerminallyFailed(obj); terminal { | |||
| return fmt.Errorf("%wHealthUtil: Job \"%v\" has failed terminally: %s", engine.ErrFatalExecution, obj.Name, msg) | |||
There was a problem hiding this comment.
Here, we use engine.ErrFatalExecution marker to denote health errors that are likely not recoverable. The caller may or may not act upon this information.
There was a problem hiding this comment.
The only thing I don't like about it is that ErrFatalExecution is defined as errors.New("fatal error: ") and that fatal error: string breaks error message readability.
zen-dog
commented
Mar 17, 2020
| @@ -58,8 +60,8 @@ func (at ApplyTask) Run(ctx Context) (bool, error) { | |||
| // 4. - Check health for all resources - | |||
| err = isHealthy(applied) | |||
| if err != nil { | |||
| if fatal := isTerminallyFailed(applied); fatal != nil { | |||
| return false, fatalExecutionError(fatal, failedTerminalState, ctx.Meta) | |||
| if errors.Is(err, engine.ErrFatalExecution) { | |||
There was a problem hiding this comment.
We return a type engine.ExecutionError struct if we encountered a fatal health error.
zen-dog
commented
Mar 17, 2020
| @@ -239,16 +241,6 @@ func isHealthy(ro []runtime.Object) error { | |||
| return nil | |||
| } | |||
|
|
|||
| func isTerminallyFailed(ro []runtime.Object) error { | |||
There was a problem hiding this comment.
Arguably this is a better separation of concerns: determining whether a resource health error is terminal is better left to the HealthUtil
zen-dog
changed the title
porridge
approved these changes
Mar 18, 2020
pkg/engine/task/task_apply_test.go
Outdated
| @@ -193,5 +196,5 @@ func (k *testEnhancer) Apply(templates map[string]string, metadata renderer.Meta | |||
| type errorEnhancer struct{} | |||
|
|
|||
| func (k *errorEnhancer) Apply(templates map[string]string, metadata renderer.Metadata) ([]runtime.Object, error) { | |||
| return nil, errors.New("always error") | |||
| return nil, fmt.Errorf("%wsomething bad happens", engine.ErrFatalExecution) | |||
There was a problem hiding this comment.
Suggested change
| return nil, fmt.Errorf("%wsomething bad happens", engine.ErrFatalExecution) | |
| return nil, fmt.Errorf("%wsomething bad happens every time", engine.ErrFatalExecution) |
ANeumann82
approved these changes
Mar 18, 2020
Signed-off-by: Aleksey Dukhovniy <alex.dukhovniy@googlemail.com>
Correct. Thought fatally failing a plan, because an API request timeout is less than ideal too. Caching for the rescue! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
previously any error returned by the
Enhancerwas treated as fatal. It made sense because it would act deterministically on a prepared set of resources. However, since #1319 enhancer also uses the discovery interface to determine whether or not the given resource is namespaced. These API requests may fail and must be retried. With this fix,DefaultEnhancerclearly marks certain errors as fatal.Fixes #1413
Signed-off-by: Aleksey Dukhovniy alex.dukhovniy@googlemail.com