Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

job-info: get internal representation of jobspec #5411

Closed
chu11 opened this issue Aug 25, 2023 · 2 comments
Closed

job-info: get internal representation of jobspec #5411

chu11 opened this issue Aug 25, 2023 · 2 comments
Assignees

Comments

@chu11
Copy link
Member

chu11 commented Aug 25, 2023

flux job info <JOBID> jobspec gets the jobspec from the KVS, but this may not be the jobspec used internally by the job-manager or is represented inside job-list.

The jobspec can be updated via jobtap plugins (perhaps adding defaults) or users can update via the future flux update command #5409.

It would be useful if flux job info <JOBID> jobspec got the presently used / represented jobspec.

Perhaps a --kvs option could be used to get what is in the KVS, similar to the --original option (and add to the Python equivalent convenience functions).

Also, an API mechanism to get this would be useful, as the flux-update command could use such a convenience function.

Implementation note: job-list already builds an internal representation of jobspec given jobspec-update events in the eventlog. It could be used to retrieve the internal representation of a jobspec. The only minor downside is that it can be racy, as jobspec-update events are sent to it via the journal. We probably don't want to create a service to retrieve the internal jobspec from job-manager.

Edit: I guess job-info could build the proper representation of jobspec since it has to read eventlog for a security check anyways.

@grondo
Copy link
Contributor

grondo commented Aug 25, 2023

As a side note, a similar treatment for R might be nice once we support resource-update events, so that flux job info JOBID R returns any updated expiration at the time of the request.

@chu11
Copy link
Member Author

chu11 commented Sep 1, 2023

So pondering this a bit more and I see multiple approaches:

  • get the redacted jobspec via job-list

    • Pro: it's already in there
    • Con: we don't have guest access check in there
    • Con: minor eventual consistency raciness
  • get the redacted jobspec via job-info

    • Pro: guest access checks already there
    • Pro: we already have to read the eventlog for guest access, so no extra KVS reads
    • Con: gotta build the jobspec up, putting cycles on job-info in broker
  • get the redacted jobspec via job-manager

    • Pro: it's already there
    • Con: we gotta support some user facing service thingie to get it from here along w/ access checks
  • create library util functions to do this for job-info / tools so we atleast have it in one place

    • Pro: no extra load on broker
    • Con: more work

No obvious winner, pros and cons each direction. Am leaning to just create a helper function that flux job info and flux update and anything in the future can all use, since it doesn't appear we need widespread use at the moment.

Edit: Thinking about this even more I'm thinking of being even lazier. Perhaps until there's a need, we should just implement something into flux job info just for flux job info. A utility function in the future can be created when needed. flux job info has the C implementation and flux update has the Python implementation for the time being.

@chu11 chu11 self-assigned this Sep 1, 2023
chu11 added a commit to chu11/flux-core that referenced this issue Sep 6, 2023
Problem: Internally within flux, the jobspec can be updated through
jobspec-update events.  When the user runs "flux job info" to get
the jobspec, it is getting the jobspec stored in the KVS and it may
not represent what is actually used.  In addition, the jobspec returned
from "flux job info" will be inconsistent to the information returned
from "flux jobs".

Solution: When retrieving the jobspec, also retrieve the job eventlog.
Internally update the jobspec through "jobspec-update" events and output
the final result to give the user a representation of what the jobspec
looks like within flux.

Add a --noupdate option to perform the old behavior, which is to just get
the jobspec stored in the KVS.

Fixes flux-framework#5411
chu11 added a commit to chu11/flux-core that referenced this issue Sep 7, 2023
Problem: Internally within flux, the jobspec can be updated through
jobspec-update events.  When the user runs "flux job info" to get
the jobspec, it is getting the jobspec stored in the KVS and it may
not represent what is actually used.  In addition, the jobspec returned
from "flux job info" will be inconsistent to the information returned
from "flux jobs".

Solution: When retrieving the jobspec, also retrieve the job eventlog.
Internally update the jobspec through "jobspec-update" events and output
the final result to give the user a representation of what the jobspec
looks like within flux.

Add a --base option to perform the old behavior, which is to just get
the jobspec stored in the KVS.

Fixes flux-framework#5411
chu11 added a commit to chu11/flux-core that referenced this issue Sep 8, 2023
Problem: Internally within flux, the jobspec can be updated through
jobspec-update events.  When the user runs "flux job info" to get
the jobspec, it is getting the jobspec stored in the KVS and it may
not represent what is actually used.  In addition, the jobspec returned
from "flux job info" will be inconsistent to the information returned
from "flux jobs".

Solution: When retrieving the jobspec, also retrieve the job eventlog.
Internally update the jobspec through "jobspec-update" events and output
the final result to give the user a representation of what the jobspec
looks like within flux.

Add a --base option to perform the old behavior, which is to just get
the jobspec stored in the KVS.

Fixes flux-framework#5411
chu11 added a commit to chu11/flux-core that referenced this issue Sep 8, 2023
Problem: Internally within flux, the jobspec can be updated through
jobspec-update events.  When the user runs "flux job info" to get
the jobspec, it is getting the jobspec stored in the KVS and it may
not represent what is actually used.  In addition, the jobspec returned
from "flux job info" will be inconsistent to the information returned
from "flux jobs".

Solution: When retrieving the jobspec, also retrieve the job eventlog.
Internally update the jobspec through "jobspec-update" events and output
the final result to give the user a representation of what the jobspec
looks like within flux.

Add a --base option to perform the old behavior, which is to just get
the jobspec stored in the KVS.

Fixes flux-framework#5411
@mergify mergify bot closed this as completed in f3da2db Sep 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants