Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make -setup load the Beat dashboards #3506

Merged
merged 1 commit into from Feb 3, 2017

Conversation

Projects
None yet
4 participants
@tsg
Copy link
Collaborator

commented Jan 31, 2017

With this change, filebeat -setup loads the sample Kibana dashboards,
being the rough equivalent of running: scripts/import_dashboards && filebeat.
This works for all Beats, not only Filebeat. The import_dashboards program
is kept for now, to avoid braking compatibility. Code wise, the two share
most of the code, which was separated in a sub-package.

All flags available for the import_dashboard are available as go-ucfg options.
For example, to load the dashboards from a directory you can do:

filebeat -e -setup -E "dashboards.dir=_meta/kibana"

Or do load the snapshot version of the dashboards:

filebeat -e -setup -E "dashboards.snapshot=true"

This also changes the pipeline loading to happen automatically regardless of
whether -setup was used or not.

Part of #3159.

TODOs:

  • Add tests for the dashboard sub-package
  • Document the -setup flag and dashboard settings
  • Changelog + deprecation notice for the import_dashboard program (?)

@tsg tsg requested a review from monicasarbu Jan 31, 2017

@tsg tsg referenced this pull request Jan 31, 2017

Closed

Meta ticket: Filebeat modules #3159

22 of 22 tasks complete

@tsg tsg force-pushed the tsg:filebeat_modules_dashboards branch 2 times, most recently from 164700a to 0d8d8b6 Feb 1, 2017

@tsg tsg requested a review from dedemorton Feb 2, 2017

@tsg

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 2, 2017

@dedemorton I'd appreciate your review on the docs part.

@tsg tsg force-pushed the tsg:filebeat_modules_dashboards branch from 0d8d8b6 to dd7f882 Feb 2, 2017

#dashboards.file:

# If this option is enabled, the snapshot URL is used instead of the default URL.
#dashboard.snapshot: false

This comment has been minimized.

Copy link
@ruflin

ruflin Feb 2, 2017

Collaborator

Nice.

This comment has been minimized.

Copy link
@tsg

tsg Feb 2, 2017

Author Collaborator

That's a feature we already have in `import_dashboards" :)

@tsg tsg removed the in progress label Feb 2, 2017

@ruflin
Copy link
Collaborator

left a comment

It's really great to have all dashboard config options now also directly in the config file and indirectly available through the -E flag.

Question for the -setup flag: If I run now ./filebeat -setup will filebeat stop after the setup phase or will it continue running? And will it load only dashboards or also pipelines, index patterns etc?

}

var (
printVersion = flag.Bool("version", false, "Print the version and exit")
setup = flag.Bool("setup", false, "Load the sample Kibana dashboards")

This comment has been minimized.

Copy link
@ruflin

ruflin Feb 2, 2017

Collaborator

Load sampel Kibana dashboards, index template, patterns?

This comment has been minimized.

Copy link
@tsg

tsg Feb 2, 2017

Author Collaborator

At the moment it only loads the dashboards, see the other answer.

@@ -0,0 +1,535 @@
package dashboards

This comment has been minimized.

Copy link
@ruflin

ruflin Feb 2, 2017

Collaborator

Why is it dashboards/dashboards/...?

This comment has been minimized.

Copy link
@tsg

tsg Feb 2, 2017

Author Collaborator

First dashboards is the current import_dashboards main package. The second is where I gathered most of the code. After we remove import_dashboards we can move the package one level up.

@@ -0,0 +1,294 @@
########################## Metricbeat Configuration ###########################

This comment has been minimized.

Copy link
@ruflin

ruflin Feb 2, 2017

Collaborator

Wasn't this under .gitignore? Do we need it again?

This comment has been minimized.

Copy link
@tsg

tsg Feb 2, 2017

Author Collaborator

Hmm, weird, i'll check.

This comment has been minimized.

Copy link
@tsg

tsg Feb 2, 2017

Author Collaborator

There was a typo in .gitignore, I deleted it again.

@tsg

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 2, 2017

Question for the -setup flag: If I run now ./filebeat -setup will filebeat stop after the setup phase or will it continue running? And will it load only dashboards or also pipelines, index patterns etc?

It will continue in case of success and stop in case of errors. Since we still have the import_dashboards script that does only that, I thought it would be interesting to experiment with letting it run in case of success. So far I like it like this, but it's debatable.

At the moment, -setup loads strictly the dashboards. Everything else (templates, pipelines, etc.) is happening automatically always, since they are lightweight operations and the Beats mallfunction if they are not loaded. The dashboards, on the other hand, are relatively heavy to load and can also be loaded after starting the Beat, without any implication regarding the correctness of the data.

@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Feb 2, 2017

@tsg Thanks for the details. My preferred option would be as following:

  • -setup loads everything (dashboards, pipelines, templates, ...) and stops afterwards as this is run only on one machine / separate machine. This would mostly replace the dashboard script.
  • Without setup, template, pipelines are loaded by default, but can be disabled.

But if -setup only loads the dashboards, I can see that you're behaviour is preferrable as we also have "only" the script.

@tsg

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 2, 2017

We could have both -setup and -setup-only as a compromise? The -setup is great for demos and getting started, since it removes one step. I didn't add -setup-only yet because we still have import_dashboards.

@ruflin

This comment has been minimized.

Copy link
Collaborator

commented Feb 2, 2017

For demos what can be done now is ./filebeat -E dashboards.enabled:true which loads everything so not even -setup flag has to be used. That is the great thing about having it in the config file. Having this, -setup-only could be -setup.

@tsg

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 2, 2017

We're splitting hairs now, but I think -setup would work better in demos & getting started than -E dashboards.enabled=true. We could have -E dashboards.stop_after_loading=true or something like that instead of -setup-only. In other words, flip when we need the -E.

Or maybe I'm wrong and we could skip both -setup and -setup-only and only rely on -E flags, this way we teach the user early on how to do that.

@tsg tsg force-pushed the tsg:filebeat_modules_dashboards branch from dd7f882 to f79a78d Feb 2, 2017

#dashboards.index:

# If enabled, load only the dashboards and not the index pattern.
#dashboards.only_dashboards: false

This comment has been minimized.

Copy link
@monicasarbu

monicasarbu Feb 2, 2017

Contributor

I think it doesn't make sense to migrate these two options (only_dashboards and only_index) from import_dashboards, and leave -setup to import both dashboards and the index pattern.

This comment has been minimized.

Copy link
@tsg

tsg Feb 2, 2017

Author Collaborator

I removed the two options from the configuration files and docs, but they still exist in the code so that they keep working for import_dashboards. We can remove them from the code as well when we remove import_dashboards.

@tsg tsg force-pushed the tsg:filebeat_modules_dashboards branch from f79a78d to 90ee353 Feb 2, 2017

Make -setup load the Beat dashboards
With this change, `filebeat -setup` loads the sample Kibana dashboards,
being the rough equivalent of running: `scripts/import_dashboards && filebeat`.
This works for all Beats, not only Filebeat. The `import_dashboards` program
is kept for now, to avoid braking compatibility. Code wise, the two share
most of the code, which was separated in a sub-package.

All flags available for the `import_dashboard` are available as go-ucfg options.
For example, to load the dashboards from a directory you can do:

    filebeat -e -setup -E "dashboards.dir=_meta/kibana"

Or do load the snapshot version of the dashboards:

    filebeat -e -setup -E "dashboards.snapshot=true"

This also changes the pipeline loading to happen automatically regardless of
whether `-setup` was used or not.

Part of #3159.

Contains updates for the docs and changelog.

@tsg tsg force-pushed the tsg:filebeat_modules_dashboards branch from 90ee353 to e10fbaa Feb 2, 2017

@monicasarbu

This comment has been minimized.

Copy link
Contributor

commented Feb 2, 2017

I think what we should improve here on longer term is to remove the necessity of -setup to load the dashboards, and have each Beat check if the dashboards version X are loaded and load them only when needed. And maybe have a "dashboards.overwrite" boolean option. This will also solve the first experience of the user as only one Beat will be able to load the dashboard, when starting more Beats at the same time.

Another improvement that we can do is to load only the Kibana dashboards for which the modules are enabled, but this requires us to split the dashboards per module. The index pattern will be loaded for all modules.

For now, I would say to keep -setup import only the dashboards (and stop afterwards), and try to remove it with the time.

@dedemorton

This comment has been minimized.

Copy link
Contributor

commented Feb 3, 2017

@tsg sorry I didn't get to the review today (didn't see your request until late in the day). I'll look in the morning. But I have a couple of initial questions:

  1. Since this is Beta level, I'm assuming we'll leave the steps for loading the dashboards as they are in the getting started guides for now, right? At what point will the new way become the preferred way to install dashboards?
  2. What about the use case where community Beats developers need to import dashboards (documented here). Will we continue to provide the import_dashboards command for that specific use case?
@tsg

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 3, 2017

Since this is Beta level, I'm assuming we'll leave the steps for loading the dashboards as they are in the getting started guides for now, right? At what point will the new way become the preferred way to install dashboards?

Correct, we don't have to change the getting started guide yet. I think the new way will become preferred in 6.0, but until then more changes might happen.

What about the use case where community Beats developers need to import dashboards (documented here). Will we continue to provide the import_dashboards command for that specific use case?

For the moment we keep the import_dashboards script, so we don't have to adjust the documentation. It'll also probably change with 6.0.

@monicasarbu monicasarbu merged commit 0d64737 into elastic:master Feb 3, 2017

4 checks passed

CLA Commit author is a member of Elasticsearch
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
default Build finished.
Details

@dedemorton dedemorton referenced this pull request Feb 14, 2017

Closed

Beats Doc Updates for 5.3 #3539

12 of 12 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.