feat(clouddriver): Basic support for retryable kato tasks #3162
Conversation
robzienert
requested review from
cfieber,
srekapalli,
dreynaud and
marchello2000
|
Thanks I Love It |
| @@ -48,10 +50,12 @@ class MonitorKatoTask implements RetryableTask { | |||
|
|
|||
| private final Clock clock | |||
| private final Registry registry | |||
| @Autowired private KatoService kato | |||
There was a problem hiding this comment.
could you add this to the constructor instead
There was a problem hiding this comment.
yeeeeeahh... I tried once and wound up reverting it. I'll try again.
| this(registry, Clock.systemUTC()) | ||
| this.kato = kato |
There was a problem hiding this comment.
this seems wrong without KatoService as a constructor arg
| import com.netflix.spinnaker.orca.pipeline.model.Stage | ||
| import groovy.transform.CompileStatic | ||
| import groovy.transform.TypeCheckingMode | ||
| import io.github.resilience4j.retry.RetryConfig |
There was a problem hiding this comment.
I don't see this used anywhere in here, old?
orca-web/config/orca.yml
Outdated
| default: | ||
| maxRetryAttempts: 3 | ||
| waitDuration: 10s | ||
| enableExponentialBackoff: true |
There was a problem hiding this comment.
we talked in person but these defaults will blow orca's 60s task ack timeout which is going to cause issues/double executions, etc
|
|
||
| if (status == ExecutionStatus.TERMINAL && katoTask.status.retryable) { | ||
| kato.resumeTask(katoTask.id) | ||
| status = ExecutionStatus.RUNNING |
There was a problem hiding this comment.
probably worth at least logging something here that we are "going to retry".
You are basically hiding an error from the user and it is possible the error goes away on retry or it becomes another error... and that makes it harder to debug issues. (this is the thing we talked about some standardization in orca about error messages... someday)
There was a problem hiding this comment.
You're right. I've updated the PR to show usage of the SystemNotification stuff I was talking about.
robzienert
force-pushed
the
kato-task-retry
branch
2 times, most recently
from
6859cdb
to
2473141
Compare
robzienert
force-pushed
the
kato-task-retry
branch
from
2473141
to
5c0816a
Compare
Depends on spinnaker/kork#383