Add nats dashboard

[ChrsMark]

This PR adds an overview dashboard for nats module as proposed on #10071.

Co-Authored-By: Stamatis Katsaounis katsaouniss@gmail.com
Co-Authored-By: Michael Katsoulis michaelkatsoulis88@gmail.com

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[ChrsMark]

@ruflin here is something we created.

TBH I'm not confident enough about the beauty of the dashboard, but this is the overview we find meaningful for our purposes.

[ruflin]

Thanks, this is great. I'm good with the dashboards as is but still need to test it on my end.

Some suggestions:

• Should we use one box per number instead of multiple in one box?
• It seems Kibana does not know it's a byte value. Probably we are missing format: bytes somewhere in the fields.yml. If we have this in, Kibana would convert the value to MB / KB etc. which would be nicer I think.

[ChrsMark]

Nice! Will update it according to the suggestions!

PS: @ruflin to make your life easier with manual testing:
you can spin up a nats container and feed it will artificial traffic using nats benchmarks (https://github.com/nats-io/go-nats/tree/master/examples/nats-bench) written in Go lang. I run the benchmark as: ./nats-bench -np 50 -ns 50 -n 1000000 -ms 16 16 foo in order to produce significant traffic. Feel free to ping if you have trouble with this.

[sayden]

Hi @ChrsMark 🙂 Amazing contribution, dashboards are very important for each module.

I think the dashboard wasn't exported correctly. I get this when trying to open it:

Also, I cannot open each visualization. It just hangs when I clicked them, first time I see something similar.

Can you tell us how did you export the dashboard?

Thanks!

[ChrsMark]

Hmm, this is an error on my end. I had the visualizations already in Kibana from previous step so when I reloaded the dashboard with make import-dashboards everything was fine.

Let me tune the visualization IDs.

@sayden, I followed the steps below:

1. ./metricbeat export dashboard -id 7fea2930-478e-11e7-b1f0-cb29bac6bf8b >> Metricbeat-nats.json
2. I had to remove some stuff like searchSourceJSON fields and replace some machine created IDs with named ones.
3. make import-dashboards

Is this the normal workflow?

[ChrsMark]

Finally I made it work smoothly with:
MODULE=nats ID=c359a6c0-1e45-11e9-a1b4-79a7ae42ab61 mage exportDashboard,
@sayden

Change regarding fields (conversion to bytes) will be addressed on another commit. (cc: @skatsaounis, @MichaelKatsoulis )

[jsoriano]

@ChrsMark this is looking great 🙂

As an extra suggestion, I'd propose to use more line/area graphs and less gauges and numbers, for example for the number of consumers we could have a graph like the one for subscriptions. Also, for the volume of messages we could have graphs with two scales, one for number of messages and another one for its size in bytes, and something similar for cpu usage.

I think gauges are great for dashboards that are not usually operated, like the ones that can be seen in a screen in the wall 🙂 But when operating or investigating an incidence in an specific period of time I find graphs with the evolution of the values much more useful.

[ChrsMark]

I think most comments addressed!

[ruflin]

As #10249 was merged this will require a rename of the dir 6 to 7.

[sayden]

Awesome @ChrsMark The chart can be loaded already.

As a general workflow, a user will load the dashboard by doing the setup with metricbeat setup. All dashboards are loaded into Kibana with this command. This is the workflow that must be tested to ensure everything is working when importing the dashboard.

Few minor things left that I didn't see/realized before (sorry):

• A description on the dashboard is missing.
• The names of the visualizations don't follow naming conventions. For example NATS IO should be IO [Metricbeat NATS]. Nats Memory should be Memory [Metricbeat NATS]. Take a look at the rest of visualization charts to see more examples. You did it correctly with the dashboard 👍

Finally, but I'm not fully sure why, some index patterns are missing. I don't have any good suggestion:

• Check that the metricbeat binary was updated before exporting.
• Check that all fields.yml files are in sync and updated before exporting.
• The script you used to export is the good one. Just in case there's another script to update here: https://github.com/elastic/beats/blob/master/dev-tools/cmd/dashboards/export_dashboards.go

@ruflin do you have any idea of why some of the index pattern fields have not been uploaded?

[sayden]

Overall looks very good. Left some comments

[sayden]

To fix the conflicts you'll need to rebase over master and then push the changes in those files after running make update

[skatsaounis]

Hi all,

I have made the rebase locally and also worked on your comments.

Tomorrow I am going to push a new patch which also contains changes to some field types (eg. long instead of integer etc) based on NATS output.

[skatsaounis]

Hi all,

I rebased to today's master branch and also updated some fields and the dashboard as well.

Below, you can see a screenshot with the new changes:

[skatsaounis]

@ruflin Hi Travis is green. Can you trigger Jenkins as well? :)

[ruflin]

jenkins, test this

[ruflin]

Tested this locally and seems to work as expected. Unfortunately my graphs don't look as nice as I don't that nice data ;-)

[ruflin]

@skatsaounis Any chance you could open a follow up PR to the docs with the above dashboard images so it shows up here? https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-module-nats.html Would be nice to show to users in advance what kind of dashboard they get.

[skatsaounis]

Hi @ruflin. Thank you for merging this PR. I am going to update the docs; no problem at all :) I will mention you to the new PR when it will be ready.