New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve and extend template and dashboard configurations #3921

Open
ruflin opened this Issue Apr 5, 2017 · 4 comments

Comments

Projects
None yet
3 participants
@ruflin
Collaborator

ruflin commented Apr 5, 2017

In 6.0 the template configuration changes as it is dynamically loaded. Also recently dashboards configs to load dashboards were introduced. In addition logic to load certain data on startup like pipelines were added. This issue propose a potential way forward with and extensible config format for the above cases.

Current configuration

The current configuration for dashboard and template looks as following:

dashboard:
  enabled: false
  url:
  directory:
  file:
  snapshot: false
  snapshot_url
  beat: metricbeat
  kibana_index: .kibana
  index:
elasticsearch:
  template:
    enabled: true
    name: "metricbeat"
    fields: "${path.config}/fields.yml"
    overwrite: false

The template part is inside elasticsearch output as it currently only applies to the ES output.

Proposal

The proposal is to place all setup related configuration options under one namespace called setup and move all config options there. This would look as following:

setup:
  fields: "${path.config}/fields.yml"
  dashboard:
    ...
  template:
    ...

The fields path is on the top level as it can not only apply to templates as it does now, but also can be used to dynamically generate the index pattern for kibana dashboards or potential pipelines in the future.

This approach would have a few advantages:

  • Setup configs are decoupled from all the other options. Potentially an extra tool (beats-tool) could make use of it.
  • Template is not directly related to the elasticsearch output. Potentially in the future the template can also be loaded through LS. It would also decouple the code.
  • An elasticserach output could be added (similar to monitoring) which inherits from the main es output, but could for example have different credentials with different access rights (load template).
  • More outputs like kibana could be added for example for the dashboards
  • More options can be added under setup like pipeline to define a list of pipelines which should be loaded

The above is a breaking change and we would need to have a migration script if possible.

Further ideas

The above change will allow to solve some more problem in #3654. These changes are not tied to the above but it would be good to do them at the same time.

  • Create a local template.json file
  • Add settings to the template

For this the config could be extended as following:

setup:
  fields: "${path.config}/fields.yml"
  beat.version: 6.2.3
  dashboard:
  template:
    enabled: true
    name: "metricbeat"
    overwrite: false
    # Defines if template should be loaded to elasticsearch
    load: true
    # Creates the json format of the template for manual loading
    output: metricbeat.template.json
    # Settings is a dynamic dict. These fields will be directly added to the template -> no logic check
    settings:
      number_of_shards: 1
      number_or_replicas: 1

In the above the follwing changed:

  • load config option was added to be able to only generate the local file and not load it into elaticsearch
  • output config option was added to define the file the template should be written to. Probably other name needed to not confuse with output config.
  • beat.version config option was added. This would be need if someone wants to load a template for a different version. Thinking of beats-tool which is not updated that often.
  • settings config option was added. This could be a dict that is added to the template to provide the possibility to configure shards / replicas and others directly when generating the template.

Related to #3654 and #3691

@tsg

This comment has been minimized.

Collaborator

tsg commented Apr 6, 2017

SGTM.

load config option was added to be able to only generate the local file and not load it into elaticsearch
output config option was added to define the file the template should be written to. Probably other name needed to not confuse with output config.

Perhaps we could reduce these to a single option (output or output_to_file) which implies load: false. Otherwise, people might get confused between enabled and load.

settings config option was added. This could be a dict that is added to the template to provide the possibility to configure shards / replicas and others directly when generating the template.

The question here is if we want to allow overwriting any mapping setting or we explicitly add things? I'd say we start with explicit settings for the things we know are useful (number of shards, number of replicas, index numerics).

@tsg

This comment has been minimized.

Collaborator

tsg commented Apr 10, 2017

Just to have mentioned it, this will make setting up the dahsboards URL from the CLI a bit more annoying: -E setup.dashboards.url=... vs the current -E dashboards.url=.... I think it's not a big deal, but just so that we're aware about this.

I also noticed that it's dashboard (singular) in your examples. Is that intentional? Because there generally are multiple dashboards per Beat, right?

@exekias

This comment has been minimized.

Member

exekias commented Apr 11, 2017

I like the idea, kudos!

@ruflin

This comment has been minimized.

Collaborator

ruflin commented Apr 19, 2017

I will start the migration step by step.

For the longer -E option: In case this becomes an issue, we could introduce something like -dashboard.url=... as a flag but I think specifying the url is more an expert or testing use case.

For dashboards: Agree that it should be plural as it is now.

ruflin added a commit to ruflin/beats that referenced this issue Apr 19, 2017

Change `dashboards.*` config options to `setup.dashboards.*`
The dashboards config options were moved under the new setup namespace. This allows more flexibility in the future together with template loading. For more details see elastic#3921.

ruflin added a commit to ruflin/beats that referenced this issue Apr 19, 2017

Change `dashboards.*` config options to `setup.dashboards.*`
The dashboards config options were moved under the new setup namespace. This allows more flexibility in the future together with template loading. For more details see elastic#3921.

monicasarbu added a commit that referenced this issue Apr 20, 2017

Change `dashboards.*` config options to `setup.dashboards.*` (#4050)
The dashboards config options were moved under the new setup namespace. This allows more flexibility in the future together with template loading. For more details see #3921.

ruflin added a commit to ruflin/beats that referenced this issue Apr 24, 2017

Change template loading to be part of the setup
* Change config option to setup.template.* from outputs.elasticsearch.template.*
* Move loading logic into template package
* Remove template loading logic from elasticsearch output
* Changelog updated
* Template tests were moved from output to template package
* Documentation was updated. Will need some more work for which a follow up Github issue will be created.

This is only migration the existing options. New options like outputting to a json file or load additional config options will be added in a follow up PR.

Questions:
* Callback logic in ES client is not used anymore. Should we remove it?

Part of elastic#3654 and elastic#3921

TODO
* Check if the right template is loaded
* Check if the template name and index pattern are set correctly. Problem because ES is not in?

ruflin added a commit to ruflin/beats that referenced this issue Apr 24, 2017

Change template loading to be part of the setup
* Change config option to setup.template.* from outputs.elasticsearch.template.*
* Move loading logic into template package
* Remove template loading logic from elasticsearch output
* Changelog updated
* Template tests were moved from output to template package
* Documentation was updated. Will need some more work for which a follow up Github issue will be created.
* Add `GetVersion()` to elasticsearch client.

This is only migration the existing options. New options like outputting to a json file or load additional config options will be added in a follow up PR.

Questions:
* Callback logic in ES client is not used anymore. Should we remove it?

Part of elastic#3654 and elastic#3921

TODO
* Check if the right template is loaded
* Check if the template name and index pattern are set correctly. Problem because ES is not in?

ruflin added a commit to ruflin/beats that referenced this issue Apr 25, 2017

Change template loading to be part of the setup
* Change config option to setup.template.* from outputs.elasticsearch.template.*
* Move loading logic into template package
* Remove template loading logic from elasticsearch output
* Changelog updated
* Template tests were moved from output to template package
* Documentation was updated. Will need some more work for which a follow up Github issue will be created.
* Add `GetVersion()` to elasticsearch client.

This is only migration the existing options. New options like outputting to a json file or load additional config options will be added in a follow up PR.

Questions:
* Callback logic in ES client is not used anymore. Should we remove it?

Part of elastic#3654 and elastic#3921

ruflin added a commit to ruflin/beats that referenced this issue Apr 28, 2017

Change template loading to be part of the setup
* Change config option to setup.template.* from outputs.elasticsearch.template.*
* Move loading logic into template package
* Remove template loading logic from elasticsearch output
* Changelog updated
* Template tests were moved from output to template package
* Documentation was updated. Will need some more work for which a follow up Github issue will be created.
* Add `GetVersion()` to elasticsearch client.
* Introduce callback registration for elasticsearch output. This should be generalised later. The template loading registers only with the output client factory which means, the template is not loaded when connecting for loading dashboards, pipeline or monitoring data which is intended.

This is only migration the existing options. New options like outputting to a json file or load additional config options will be added in a follow up PR.

Part of elastic#3654 and elastic#3921

tsg added a commit that referenced this issue May 4, 2017

Change template loading to be part of the setup (#4080)
* Change config option to setup.template.* from outputs.elasticsearch.template.*
* Move loading logic into template package
* Remove template loading logic from elasticsearch output
* Changelog updated
* Template tests were moved from output to template package
* Documentation was updated. Will need some more work for which a follow up Github issue will be created.
* Add `GetVersion()` to elasticsearch client.
* Introduce callback registration for elasticsearch output. This should be generalised later. The template loading registers only with the output client factory which means, the template is not loaded when connecting for loading dashboards, pipeline or monitoring data which is intended.

This is only migration the existing options. New options like outputting to a json file or load additional config options will be added in a follow up PR.

Part of #3654 and #3921

@tsg tsg referenced this issue Jul 24, 2017

Closed

Document breaking changes in 6.0 #4737

28 of 28 tasks complete

ramon-garcia added a commit to ramon-garcia/beats that referenced this issue Dec 5, 2017

Change template loading to be part of the setup (elastic#4080)
* Change config option to setup.template.* from outputs.elasticsearch.template.*
* Move loading logic into template package
* Remove template loading logic from elasticsearch output
* Changelog updated
* Template tests were moved from output to template package
* Documentation was updated. Will need some more work for which a follow up Github issue will be created.
* Add `GetVersion()` to elasticsearch client.
* Introduce callback registration for elasticsearch output. This should be generalised later. The template loading registers only with the output client factory which means, the template is not loaded when connecting for loading dashboards, pipeline or monitoring data which is intended.

This is only migration the existing options. New options like outputting to a json file or load additional config options will be added in a follow up PR.

Part of elastic#3654 and elastic#3921

athom added a commit to athom/beats that referenced this issue Jan 25, 2018

Change `dashboards.*` config options to `setup.dashboards.*` (elastic…
…#4050)

The dashboards config options were moved under the new setup namespace. This allows more flexibility in the future together with template loading. For more details see elastic#3921.

athom added a commit to athom/beats that referenced this issue Jan 25, 2018

Change template loading to be part of the setup (elastic#4080)
* Change config option to setup.template.* from outputs.elasticsearch.template.*
* Move loading logic into template package
* Remove template loading logic from elasticsearch output
* Changelog updated
* Template tests were moved from output to template package
* Documentation was updated. Will need some more work for which a follow up Github issue will be created.
* Add `GetVersion()` to elasticsearch client.
* Introduce callback registration for elasticsearch output. This should be generalised later. The template loading registers only with the output client factory which means, the template is not loaded when connecting for loading dashboards, pipeline or monitoring data which is intended.

This is only migration the existing options. New options like outputting to a json file or load additional config options will be added in a follow up PR.

Part of elastic#3654 and elastic#3921

amomchilov pushed a commit to amomchilov/Filebeat that referenced this issue Apr 19, 2018

Change `dashboards.*` config options to `setup.dashboards.*` (#4050)
The dashboards config options were moved under the new setup namespace. This allows more flexibility in the future together with template loading. For more details see elastic/beats#3921.

amomchilov pushed a commit to amomchilov/Filebeat that referenced this issue Apr 19, 2018

Change template loading to be part of the setup (#4080)
* Change config option to setup.template.* from outputs.elasticsearch.template.*
* Move loading logic into template package
* Remove template loading logic from elasticsearch output
* Changelog updated
* Template tests were moved from output to template package
* Documentation was updated. Will need some more work for which a follow up Github issue will be created.
* Add `GetVersion()` to elasticsearch client.
* Introduce callback registration for elasticsearch output. This should be generalised later. The template loading registers only with the output client factory which means, the template is not loaded when connecting for loading dashboards, pipeline or monitoring data which is intended.

This is only migration the existing options. New options like outputting to a json file or load additional config options will be added in a follow up PR.

Part of elastic/beats#3654 and elastic/beats#3921
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment