Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/flux-job attach: add timestamps, --show-exec option #2388

Merged
merged 5 commits into from Sep 22, 2019

Conversation

@garlick
Copy link
Member

commented Sep 21, 2019

This PR adds timestamps (with t=0 normalized to the submit event) to events displayed with the flux job attach --show-events option. It also adds a new --show-exec option that displays events from the guest.exec.eventlog in the same format.

garlick added 3 commits Sep 21, 2019
Problem: attach event handling is somewhat involved and
may benefit from some inline documentation.

Add comment block for each attach continuation function,
and rename print_output_continuation to attach_output_continuation
for naming consitency.
Problem: flux-job attach has a context variable for
the result of calling optparse_hasopt ("show-events"),
but it also stores the optparse context in the same
struct.  This is unnecessary and won't scale as options
are added.

Drop ctx->show_events.
When the flux job attach --show-events option is in effect,
display a timestamp for each eventlog entry, in %.3f seconds
format.  The timestamps are normalized to give the first entry
a timestamp of 0.000.

Move the code to a helper function, print_eventlog_entry().
@garlick

This comment has been minimized.

Copy link
Member Author

commented Sep 22, 2019

Example with both event logs (sorting by timestamp since otherwise they are out of order)

$ ./flux job attach --show-exec --show-events 116685537280 2>&1 | sort -n -k2,2
job-event: 0.000 submit {"userid":5588,"priority":16,"flags":0}
job-event: 0.015 depend
job-event: 0.021 alloc {"note":"rank0/core0"}
exec-event: 0.022 init
exec-event: 0.025 starting
exec-event: 0.028 running
job-event: 0.029 start
exec-event: 0.048 output-ready
exec-event: 0.054 cleanup.start {"ranks":"all"}
exec-event: 0.054 complete {"status":0}
job-event: 0.055 finish {"status":0}
exec-event: 0.057 cleanup.finish {"ranks":"all","rc":0}
exec-event: 0.058 done
job-event: 0.061 release {"ranks":"all","final":true}
job-event: 0.062 clean
job-event: 0.062 free
Copy link
Contributor

left a comment

This looks super useful! I wonder if there is some way to get these events somewhat sorted without piping to an external sort command? Perhaps pushing entries onto a sorted zlistx before printing entries from the list in a check watcher?

This might actually be a nice little module -- consume eventlog entries from multiple sources and print every N ms in sorted order, using a registered "print" handler for each entry type...

Anyway, that is probably a separate issue.

@@ -108,6 +108,9 @@ static struct optparse_option attach_opts[] = {
{ .name = "show-events", .key = 'E', .has_arg = 0,
.usage = "Show job events on stderr",
},
{ .name = "show-exec", .key = 'G', .has_arg = 0,

This comment has been minimized.

Copy link
@grondo

grondo Sep 22, 2019

Contributor

Not even sure you need a short opt here, but would -X make more mnemonic sense than -G?

@grondo
grondo approved these changes Sep 22, 2019
Copy link
Contributor

left a comment

LGTM! Feel free to squash and set merge-when-passing.

garlick added 2 commits Sep 21, 2019
Add an attach option to display eventlog entries from the
guest.exec.eventlog, in the same format as the main
job eventlog entries.

Change the logic slightly to:
* Request the guest eventlog AFTER the main eventlog 'submit'
  event is received.  This ensures guest events do not arrive
  before the timestamp zero point can be established by submit.

* Don't cancel the guest eventlog after the 'output-ready' event.
  Continue processing guest events for display.

Note that if both --show-exec and --show-events are specified,
events from the two logs may be displayed out of time order,
especially if the logs are not being produced in real time as
they are consumed.
@garlick garlick force-pushed the garlick:job_attach_eventlog branch from 7681749 to ca0a990 Sep 22, 2019
@garlick

This comment has been minimized.

Copy link
Member Author

commented Sep 22, 2019

OK, thanks!

@mergify mergify bot merged commit 75ff077 into flux-framework:master Sep 22, 2019
2 checks passed
2 checks passed
Summary 1 rule matches
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@codecov-io

This comment has been minimized.

Copy link

commented Sep 22, 2019

Codecov Report

Merging #2388 into master will increase coverage by 0.02%.
The diff coverage is 87.5%.

@@            Coverage Diff             @@
##           master    #2388      +/-   ##
==========================================
+ Coverage   80.89%   80.91%   +0.02%     
==========================================
  Files         222      222              
  Lines       34986    34994       +8     
==========================================
+ Hits        28303    28317      +14     
+ Misses       6683     6677       -6
Impacted Files Coverage Δ
src/cmd/flux-job.c 85.32% <87.5%> (+0.46%) ⬆️
src/common/libflux/mrpc.c 87.79% <0%> (-1.19%) ⬇️
src/modules/job-exec/job-exec.c 75.49% <0%> (-0.22%) ⬇️
src/common/libflux/message.c 80.5% <0%> (-0.14%) ⬇️
src/modules/connector-local/local.c 74.41% <0%> (+1.3%) ⬆️
@garlick garlick referenced this pull request Sep 30, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.