Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Load configuration from modules.d when running setup subcommand #12340

Merged
merged 8 commits into from Jun 4, 2019

Conversation

@kvch
Copy link
Contributor

kvch commented May 29, 2019

Previously, when running the subcommand setup, configuration from modules.d was not loaded. This lead to Filebeat ignoring settings of the module system and loading incorrect pipelines.

Now when running setup configuration from the modules folder are read and pipelines are loaded.

TODO

  • more testing to check corner cases
  • add automated test
  • add changelog entry

Closes #9747

@kvch kvch requested a review from elastic/beats as a code owner May 29, 2019
filebeat/fileset/setup.go Outdated Show resolved Hide resolved
@kvch kvch force-pushed the kvch:fix-filebeat-setup-modules.d branch 2 times, most recently from 4a536de to 6d22781 May 30, 2019
kvch added 6 commits May 29, 2019
Closes #9747
@kvch kvch force-pushed the kvch:fix-filebeat-setup-modules.d branch from 6d22781 to 03a3388 May 31, 2019
@kvch kvch requested review from ruflin, faec and jsoriano and removed request for faec May 31, 2019
Copy link
Member

jsoriano left a comment

Thanks for fixing this! We have been removing the convert_timezones conditions from the pipeline templates (#12253), but I guess we should continue supporting templates in pipelines for other cases by now.

filebeat/beater/filebeat.go Outdated Show resolved Hide resolved
filebeat/beater/filebeat.go Show resolved Hide resolved
filebeat/tests/system/input/system.yml Outdated Show resolved Hide resolved
libbeat/tests/system/beat/beat.py Outdated Show resolved Hide resolved
filebeat/beater/filebeat.go Show resolved Hide resolved
Co-Authored-By: Jaime Soriano Pastor <jaime.soriano@elastic.co>
@codecov

This comment has been minimized.

Copy link

codecov bot commented Jun 4, 2019

Codecov Report

Merging #12340 into master will decrease coverage by 0.22%.
The diff coverage is 0%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #12340      +/-   ##
==========================================
- Coverage   58.02%    57.8%   -0.23%     
==========================================
  Files         933      934       +1     
  Lines       57454    57485      +31     
==========================================
- Hits        33340    33231     -109     
- Misses      21050    21192     +142     
+ Partials     3064     3062       -2
Impacted Files Coverage Δ
filebeat/fileset/setup.go 0% <0%> (ø)
libbeat/cfgfile/reload.go 0% <0%> (ø) ⬆️
filebeat/beater/filebeat.go 0% <0%> (ø) ⬆️
filebeat/fileset/pipelines.go 64.1% <0%> (-17.1%) ⬇️
libbeat/publisher/pipeline/acker.go 0% <0%> (-14.37%) ⬇️
filebeat/fileset/modules.go 36.88% <0%> (-14.35%) ⬇️
libbeat/publisher/queue/memqueue/ringbuf.go 55.96% <0%> (-5.51%) ⬇️
metricbeat/module/docker/memory/helper.go 63.15% <0%> (-5.03%) ⬇️
...etricbeat/module/rabbitmq/connection/connection.go 68.75% <0%> (-4.59%) ⬇️
metricbeat/module/rabbitmq/exchange/exchange.go 68.75% <0%> (-4.59%) ⬇️
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4319c31...f491b71. Read the comment docs.

@kvch kvch merged commit 5d7c1fa into elastic:master Jun 4, 2019
4 checks passed
4 checks passed
CLA All commits in pull request signed
Details
Hound 1 violation found.
beats-ci Build finished.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
andrewvc added a commit to andrewvc/beats that referenced this pull request Jun 12, 2019
…tic#12340)

Previously, when running the subcommand `setup`, configuration from `modules.d` was not loaded. This lead to Filebeat ignoring settings of modules which had special template directive in their pipelines. Thus, uploading incorrect pipelines.

Now when running setup configuration from the modules folder are read and pipelines are loaded.

Closes elastic#9747
@adriansr

This comment has been minimized.

Copy link
Member

adriansr commented Jul 17, 2019

@kvch as this is marked as a bug in CHANGELOG, can you backport it to 7.2? Or is there a reason not to?

@kvch

This comment has been minimized.

Copy link
Contributor Author

kvch commented Jul 17, 2019

No reason. Backporting it.

kvch added a commit to kvch/beats that referenced this pull request Jul 17, 2019
…tic#12340)

Previously, when running the subcommand `setup`, configuration from `modules.d` was not loaded. This lead to Filebeat ignoring settings of modules which had special template directive in their pipelines. Thus, uploading incorrect pipelines.

Now when running setup configuration from the modules folder are read and pipelines are loaded.

Closes elastic#9747
(cherry picked from commit 5d7c1fa)
kvch added a commit to kvch/beats that referenced this pull request Jul 17, 2019
…tic#12340)

Previously, when running the subcommand `setup`, configuration from `modules.d` was not loaded. This lead to Filebeat ignoring settings of modules which had special template directive in their pipelines. Thus, uploading incorrect pipelines.

Now when running setup configuration from the modules folder are read and pipelines are loaded.

Closes elastic#9747
(cherry picked from commit 5d7c1fa)
kvch added a commit to kvch/beats that referenced this pull request Jul 17, 2019
…tic#12340)

Previously, when running the subcommand `setup`, configuration from `modules.d` was not loaded. This lead to Filebeat ignoring settings of modules which had special template directive in their pipelines. Thus, uploading incorrect pipelines.

Now when running setup configuration from the modules folder are read and pipelines are loaded.

Closes elastic#9747
(cherry picked from commit 5d7c1fa)
kvch added a commit to kvch/beats that referenced this pull request Jul 17, 2019
…tic#12340)

Previously, when running the subcommand `setup`, configuration from `modules.d` was not loaded. This lead to Filebeat ignoring settings of modules which had special template directive in their pipelines. Thus, uploading incorrect pipelines.

Now when running setup configuration from the modules folder are read and pipelines are loaded.

Closes elastic#9747
(cherry picked from commit 5d7c1fa)
kvch added a commit that referenced this pull request Jul 17, 2019
…ing setup subcommand (#12941)

* Load configuration from modules.d when running setup subcommand (#12340)

Previously, when running the subcommand `setup`, configuration from `modules.d` was not loaded. This lead to Filebeat ignoring settings of modules which had special template directive in their pipelines. Thus, uploading incorrect pipelines.

Now when running setup configuration from the modules folder are read and pipelines are loaded.

Closes #9747
(cherry picked from commit 5d7c1fa)

* add missing function to libbeat/beat/beat.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.