Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

job-manager: allow dependencies on completed jobs #4287

Closed
garlick opened this issue Apr 15, 2022 · 2 comments
Closed

job-manager: allow dependencies on completed jobs #4287

garlick opened this issue Apr 15, 2022 · 2 comments

Comments

@garlick
Copy link
Member

garlick commented Apr 15, 2022

Some other random thoughts:

  • The dependency plugins in the job-manager could also make great use of the inactive job cache to allow jobs to have dependencies on other, completed jobs (right now jobs can only have dependencies on active jobs). This might only work if the job "result" for inactive jobs is available.
  • Apropos of the previous bullet, it might be nice to have a function to lookup a jobid that checks active hash and falls back to inactive hash. Callers can check the job state to determine if the job was active vs inactive to determine from which hash the job was found.

Originally posted by @grondo in #4286 (comment)

@garlick
Copy link
Member Author

garlick commented May 2, 2022

This seems like a good and relatively easy improvement to dependencies, so adding to the next milestone.

It seems like rfc 26 does not place any restrictions on the antecedent job state when a simple job dependency is added, so this is actually required for a correct implementation?

@garlick garlick added this to the flux-core v0.40.0 milestone May 2, 2022
@grondo
Copy link
Contributor

grondo commented May 2, 2022

Yes, but RFC 26 describes the dependency specification and I don't think covers details of how dependencies are satisfied? But I agree it would be more correct and less surprising to fix this particular issue. (Though if there is an inactive job purge, I'm guessing the same problem could be introduced, so I would avoid explicitly requiring that any inactive job can be a antecedent...)

@garlick garlick modified the milestones: flux-core v0.40.0, flux-core v0.41.0 Jun 6, 2022
grondo added a commit to grondo/flux-core that referenced this issue Jul 2, 2022
Problem: Job manager plugins can now lookup inactive jobs via the
flux_jobtap_job_lookup(3) function, but the dependency-after plugin
does not properly handle jobs in the INACTIVE state.

Add a check for a job state of INACTIVE in the dependency-after
plugin, and attempt ot either resolve the depedency immediately or
reject the job if the dependency is not satisfiable.

Fixes flux-framework#4287
grondo added a commit to grondo/flux-core that referenced this issue Jul 2, 2022
Problem: Job manager plugins can now lookup inactive jobs via the
flux_jobtap_job_lookup(3) function, but the dependency-after plugin
does not properly handle jobs in the INACTIVE state.

Add a check for a job state of INACTIVE in the dependency-after
plugin, and attempt ot either resolve the depedency immediately or
reject the job if the dependency is not satisfiable.

Fixes flux-framework#4287
@mergify mergify bot closed this as completed in a246231 Jul 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants