Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configuration refactoring #1417

Closed
tsg opened this issue Apr 19, 2016 · 1 comment

Comments

Projects
None yet
2 participants
@tsg
Copy link
Collaborator

commented Apr 19, 2016

This is a meta issue for reorganizing our default configuration files for 5.0. The problems we're trying to solve are:

  • Getting whitespace wrong in YAML is our number one source of issues on discuss. Now the Beats supports dotted notation (e.g. output.elasticsearch.hosts) in the configuration file, which greatly reduces the indentation levels required.
  • Inconsistent namespacing around Beats. Filebeat has all its options under filebeat, Topbeat has/had input, Packetbeat has several options toplevel. Correct namespacing is a prerequisite for building more Beats in the same binary/process.
  • The Libbeat options are not well organized, we have some stuff top level and some other stuff under beat.
  • Because we put all available options in the configuration file, the default files have gotten really large and hard to modify for the common options.

TODOs:

  • Rename the input section to topbeat in Topbeat. #1211.
  • Move top level Packetbeat sections (interfaces, flows, protocols, procs) under the packetbeat namespace #1490
  • Move settings from the beat namespace to top level. #1570
  • Change libbeat configuration file to use the dotted notation #1544 #1624
  • Change topbeat configuration file to use the dotted notation #1544 #1624
  • Change filebeat configuration file to use the dotted notation #1544 #1624
  • Change packetbeat configuration file to use the dotted notation #1544 #1624
  • Change winlogbeat configuration file to use the dotted notation #1544 #1624
  • Change metricbeat configuration file to use the dotted notation #1544 #1624
  • Create migration guide/tool: #1488
  • Update the system tests with the latest format of the config files. #1688
  • Create a minimized version of each config file. Make it the default. #1637 #1674 #1682
  • Add Kafka and Redis outputs to the "full" config files. #1690
  • Review all docs to update examples from configuration files. #1709

@monicasarbu monicasarbu added P1 and removed priority refactoring labels Apr 19, 2016

tsg added a commit to tsg/beats that referenced this issue Apr 20, 2016

If confdir is relative, make it relative to the config path
This should hopefully  be the least surprise for the user. Part of elastic#1417.

andrewkroh added a commit that referenced this issue Apr 20, 2016

If confdir is relative, make it relative to the config path (#1435)
This should hopefully  be the least surprise for the user. Part of #1417.

ruflin added a commit to ruflin/beats that referenced this issue Apr 26, 2016

Migrate packetbeat config options to namespace
* Fix docs by adding indentation
* Remove singleton config

Relates to elastic#1417

ruflin added a commit to ruflin/beats that referenced this issue Apr 27, 2016

Migrate packetbeat config options to namespace
* Fix docs by adding indentation
* Remove singleton config

Relates to elastic#1417

ruflin added a commit to ruflin/beats that referenced this issue Apr 27, 2016

Migrate packetbeat config options to namespace
* Fix docs by adding indentation
* Remove singleton config

Relates to elastic#1417

ruflin added a commit to ruflin/beats that referenced this issue Apr 27, 2016

Migrate packetbeat config options to namespace
* Fix docs by adding indentation
* Remove singleton config

Relates to elastic#1417

tsg added a commit that referenced this issue Apr 27, 2016

Migrate packetbeat config options to namespace (#1490)
* Fix docs by adding indentation
* Remove singleton config

Relates to #1417

tsg added a commit to tsg/beats that referenced this issue May 4, 2016

Move "beat" (ex-shipper) config settings to be top level
Part of elastic#1417. Done in a non-BWC way, because I think it's not worth
complicating the code. I'd rather have a script that migrates the
configuration.

tsg added a commit to tsg/beats that referenced this issue May 4, 2016

Move "beat" (ex-shipper) config settings to be top level
Part of elastic#1417. Done in a non-BWC way, because I think it's not worth
complicating the code. I'd rather have a script that migrates the
configuration.

tsg added a commit to tsg/beats that referenced this issue May 9, 2016

Move "beat" (ex-shipper) config settings to be top level
Part of elastic#1417. Done in a non-BWC way, because I think it's not worth
complicating the code. I'd rather have a script that migrates the
configuration.

ruflin added a commit that referenced this issue May 9, 2016

Move "beat" (ex-"shipper") config settings to be top level (#1570)
* Move "beat" (ex-shipper) config settings to be top level

Part of #1417. Done in a non-BWC way, because I think it's not worth
complicating the code. I'd rather have a script that migrates the
configuration.

* Make winlogbeat accept the new top-level settings

The list has grown quite long, switching to ucfg validation for this would be nice.
Also added explicit key names to config.

tsg added a commit to tsg/beats that referenced this issue May 12, 2016

Configuration files restyling
Little changes content wise, but some style changes:

* Use ###, ===, and --- headers to indicate hierarchy. This makes it
  easier to skim the config file. Idea stolen from elasticsearch.yml
* Unindent filebeat prospectors and metricbeat modules by one level.
  This is still valid yaml, and fewer spaces make it harder to mess up the
  white spaces.
* Reorganized a bit the logging config
* Moved the "general" libbeat section before the outputs
* Other fairly minor changes to the beats yaml files.

This is a follow up of elastic#1544 and part of elastic#1417.

ruflin added a commit that referenced this issue May 12, 2016

Configuration files restyling (#1624)
Little changes content wise, but some style changes:

* Use ###, ===, and --- headers to indicate hierarchy. This makes it
  easier to skim the config file. Idea stolen from elasticsearch.yml
* Unindent filebeat prospectors and metricbeat modules by one level.
  This is still valid yaml, and fewer spaces make it harder to mess up the
  white spaces.
* Reorganized a bit the logging config
* Moved the "general" libbeat section before the outputs
* Other fairly minor changes to the beats yaml files.

This is a follow up of #1544 and part of #1417.

* Added a new line in front of the libbeat options
* Make the styling changes for Metricbeat in automation

tsg added a commit to tsg/beats that referenced this issue May 13, 2016

Configuration files restyling
Little changes content wise, but some style changes:

* Use ###, ===, and --- headers to indicate hierarchy. This makes it
  easier to skim the config file. Idea stolen from elasticsearch.yml
* Unindent filebeat prospectors and metricbeat modules by one level.
  This is still valid yaml, and fewer spaces make it harder to mess up the
  white spaces.
* Reorganized a bit the logging config
* Moved the "general" libbeat section before the outputs
* Other fairly minor changes to the beats yaml files.

This is a follow up of elastic#1544 and part of elastic#1417.

tsg added a commit to tsg/beats that referenced this issue May 20, 2016

Updated configuration templates used in tests
The goal was to bring them closer to the new conventions we
use in the configuration files, so what we test is closer to what
we ship in terms of configuration.

I also made steps in unifying the libbeat portion of these templates so
we could remove the duplication in the future.

Part of elastic#1417.

tsg added a commit to tsg/beats that referenced this issue May 20, 2016

Updated configuration templates used in tests
The goal was to bring them closer to the new conventions we
use in the configuration files, so what we test is closer to what
we ship in terms of configuration.

I also made steps in unifying the libbeat portion of these templates so
we could remove the duplication in the future.

Part of elastic#1417.

tsg added a commit to tsg/beats that referenced this issue May 20, 2016

Updated configuration templates used in tests
The goal was to bring them closer to the new conventions we
use in the configuration files, so what we test is closer to what
we ship in terms of configuration.

I also made steps in unifying the libbeat portion of these templates so
we could remove the duplication in the future.

Part of elastic#1417.

ruflin added a commit that referenced this issue May 23, 2016

Updated configuration templates used in tests (#1688)
The goal was to bring them closer to the new conventions we
use in the configuration files, so what we test is closer to what
we ship in terms of configuration.

I also made steps in unifying the libbeat portion of these templates so
we could remove the duplication in the future.

Part of #1417.
@tsg

This comment has been minimized.

Copy link
Collaborator Author

commented May 24, 2016

All done. 😎

@tsg tsg closed this May 24, 2016

medcl added a commit to medcl/beats that referenced this issue Aug 17, 2016

Move "beat" (ex-"shipper") config settings to be top level (elastic#1570
)

* Move "beat" (ex-shipper) config settings to be top level

Part of elastic#1417. Done in a non-BWC way, because I think it's not worth
complicating the code. I'd rather have a script that migrates the
configuration.

* Make winlogbeat accept the new top-level settings

The list has grown quite long, switching to ucfg validation for this would be nice.
Also added explicit key names to config.

medcl added a commit to medcl/beats that referenced this issue Aug 17, 2016

Configuration files restyling (elastic#1624)
Little changes content wise, but some style changes:

* Use ###, ===, and --- headers to indicate hierarchy. This makes it
  easier to skim the config file. Idea stolen from elasticsearch.yml
* Unindent filebeat prospectors and metricbeat modules by one level.
  This is still valid yaml, and fewer spaces make it harder to mess up the
  white spaces.
* Reorganized a bit the logging config
* Moved the "general" libbeat section before the outputs
* Other fairly minor changes to the beats yaml files.

This is a follow up of elastic#1544 and part of elastic#1417.

* Added a new line in front of the libbeat options
* Make the styling changes for Metricbeat in automation

medcl added a commit to medcl/beats that referenced this issue Aug 17, 2016

Updated configuration templates used in tests (elastic#1688)
The goal was to bring them closer to the new conventions we
use in the configuration files, so what we test is closer to what
we ship in terms of configuration.

I also made steps in unifying the libbeat portion of these templates so
we could remove the duplication in the future.

Part of elastic#1417.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.