Allow tailing event stream for specific job from job view

[chuckyz]

This is a follow-up to #56 , it would be fantastic if we could view an event stream for a scoped job+namespace when viewing any given job. This would allow something like watching an event stream through wander during a rollout.

[robinovitch61]

Cool! Ok I might need your help determining what the most helpful behavior would be for this.

What I'm thinking is if you press lowercase e on the jobs page, you see an events stream specific to the selected job.

If you press uppercase E on the jobs page, you see a global events stream, the same way you do now for lowercase e.

The global events stream may continue to be configured via the wander_event_topics and wander_event_namespace configuration options (https://github.com/robinovitch61/wander#configuration), or may get its own specific configuration.

For the lowercase e, job-specific events stream, the namespace would be the same as the selected job, and I'm wondering what event topics should be shown.

A few I'm thinking of:

1. Only the topic Job:<selected-job> will be shown in the job-specific events stream. But I worry this might not actually show every event of interest during a rollout.

2. The set of topics configured in wander_event_topics would be stripped of their suffixes and filtered for that job. No new job-specific topic configuration option.

• example with the default value for wander_event_topics, Job,Allocation,Deployment,Evaluation, would show all the following filtered topics: Job:<selected-job>, Allocation:<selected-job>, Deployment:<selected-job>, and Evaluation:<selected-job>.
• example with some custom value for wander_event_topics, Job:my-job,Job:my-other-job,Allocation:my-job,Deployment:*, would show all the following filtered topics: Job:<selected-job>, Allocation:<selected-job>, and Deployment:<selected-job>.

3. A new config option is introduced, wander_job_specific_event_topics or something like that. It separates the global event topics config wander_event_topics from the job specific event topics

• example with the value for wander_job_specific_event_topics as Job,Allocation, would show all the following filtered topics for a specific job: Job:<selected-job>, and Allocation:<selected-job>, regardless of the value for wander_event_topics

Let me know if this behavior in general sounds good (job specific and global event tails) and which of these 3 configuration options, or if some other option, might be best for the job-specific event tail. You could possibly examine the shown events in wander today by setting wander_event_topics to just Job:<desired-job>,Allocation:<desired-job> etc. and see the events that show up in the global view for that rollout, or even just play with it via curl as in the sample commands here.

[chuckyz]

What I'm thinking is if you press lowercase e on the jobs page, you see an events stream specific to the selected job.

Yes please!

If you press uppercase E on the jobs page, you see a global events stream, the same way you do now for lowercase e.

Sounds perfect

One small request is to please make the default namespace '*' instead of 'default.' The way that Nomad handles ACL tokens and namespaces, '*' is kind of special in that you get access to every namespace that the ACL token can see. (at least last time I checked, I could be wrong!) I think that's pretty good behavior.

I think you are correct in that option 1. is not that useful. Option 2 is almost always what a user might want (I know it's what I want), where you can watch a job get registered, all the old allocs die, new ones get spawned in, and any evals that are triggered from various things happening and watch the deployment as it progresses.

I think 2 is better than 3 from my POV, and 1 is unfortunately not too useful.

Something that would also be useful -- if it's not too much work -- is the event stream on a single alloc. Sometimes you get an alloc that just gets hella cranky and being able to go 'alright, what are you, and only you doing.' I'd imagine that'd be e on a single alloc from within a job view.

I do the curl thing a lot sometimes. We actually run a python script which tails the global event stream from all namespaces and dumps it into Loki, but it's a lot of events and sometimes having a real-time stream at your fingertips is just what you need.

[robinovitch61]

Awesome, thank you for the feedback! I'll look into the default * namespace and get the jobs-specific event filtering with Option 2 implemented (that one also makes the most sense to me).

I'll probably file the allocation event stream as a follow up, but seems like it will be doable based on this example from the docs:

--data-urlencode "topic=Node:ccc4ce56-7f0a-4124-b8b1-a4015aa82c40" \

[robinovitch61]

Quick follow-up - I just tried * by setting the following in my ~/.wander.yaml:

wander_event_namespace: "*"

And unfortunately saw a "Permission denied" response that I don't see when I use default. So either this behavior:

The way that Nomad handles ACL tokens and namespaces, '*' is kind of special in that you get access to every namespace that the ACL token can see. (at least last time I checked, I could be wrong!) I think that's pretty good behavior.

is the default behavior in later nomad versions only, or it's not the default behavior :(. Agreed it would be sweet if it was.

Edit: from the nomad docs:

If you specify all namespaces * you'll either need a management token, or an ACL Policy that explicitly applies to all namespaces *.

Are you ok with just manually setting --event-namespace, wander_event_namespace in yaml, or WANDER_EVENT_NAMESPACE env variable to * in your own environment? I'd rather err on the side of fewer permissions errors even if that means fewer results by default. I'll make sure to make it obvious in the events pages header which namespace is being shown.

[robinovitch61]

Alright, this is released in v0.5.0! Let me know how it goes

[chuckyz]

facepalm! This works fantastic. '*' not working is what I get for not paying attention in my environment vs. others!