New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Execute dependent job when all dependencies finished. #382
Conversation
Summary: tbd JIRA issues: DCOS_OSS-5974
if (dependenciesState.shouldTriggerJob(lastSuccessfulRun)) { | ||
log.info(s"Start next run of job ${spec.id}, all parents finished successfully") | ||
runService.startJobRun(initSpec).pipeTo(self) | ||
context.become(running) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I told @pierrebeitz already what limitation we have with this.
First, I think we need to handle the result of startJobRun
. It stands to question whether we need another receive method for that.
Second and this is more important, I saw job runs triggered each time there was an update. Imagine C run five seconds depends on A and B. B is run every ten seconds but A every ten millis. This would trigger C multiple times.
A way to avoid this is to save the lastCreatedAt
for C and use this as well to check whether we already scheduled a run for C or not. I will create a follow up as a proposal.
class JobSpecDependencyActor(initSpec: JobSpec, runService: JobRunService) extends Actor with Stash with ActorLogging { | ||
|
||
private[impl] var spec = initSpec | ||
private[impl] var lastSuccessfulRun: Instant = Instant.MIN |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should load the last job run as a failover pattern?
Summary:
This the business logic of MIP 0008 "Jobs with Dependencies". We had an actor analog
to the
JobSpecSchedulerActor
. This new actor subscribes to all job run events and updatesan internal state. Once all parents have a successful run that is new than the child run we
trigger new job execution.
JIRA issues: DCOS_OSS-5974