Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the option to generate the template into a file #4323

Merged
merged 4 commits into from May 18, 2017

Conversation

Projects
None yet
3 participants
@tsg
Copy link
Collaborator

commented May 16, 2017

Part of #3654.

This adds two settings: setup.template.output_to_file.path and setup.template.output_to_file.version. The version refers to the ES version and is optional, we'll use the Beats version if not specified. I put it under output_to_file to make it clear that it only applies when outputting to a file:

To generate a config, one can do:

./metricbeat -e -E setup.template.output_to_file.path=template.json

In the current version, the Beat automatically stops after generating the template, but the output might be slightly confusing:

2017/05/16 09:55:02.043671 load.go:116: INFO Template for Elasticsearch 6.0.0-alpha2 written to: template.json
2017/05/16 09:55:02.043717 beat.go:538: CRIT Exiting: Stopping after successfully writing the template to the file.
Exiting: Stopping after successfully writing the template to the file.

IMO this is better than the alternative of leaving it running.

To generate the template for the 2.x version, one can do:

./metricbeat -e -E setup.template.output_to_file.path=template.json -E setup.template.output_to_file.version=2.4.0

Remaining TODOs:

  • System test
  • Docs
  • Changelog
@ruflin

This comment has been minimized.

Copy link
Collaborator

commented May 16, 2017

Perhaps we should discuss again with after --setup the beat should stop? Because then one could do:

./metricbeat --setup -e -E setup.template.output_to_file.path=template.json

Or assuming -S is implies --setup and prefixes -E with the setup namespace. Then the following could be used:

./metricbeat -S template.output_to_file.path=template.json

// XXX: Should we kill the Beat here or just continue?
return fmt.Errorf("Stopping after successfully writing the template to the file.")

This comment has been minimized.

Copy link
@elasticmachine

elasticmachine May 16, 2017

Collaborator

[golint] reported by reviewdog 🐶
error strings should not be capitalized or end with punctuation or a newline

This comment has been minimized.

Copy link
@tsg

tsg May 16, 2017

Author Collaborator

In this case, dear reviewdog, we know that the error is final, so the punctuation looks better to the user.

@@ -80,6 +82,41 @@ func (l *Loader) Load() error {
return nil
}

func (l *Loader) Generate() error {

This comment has been minimized.

Copy link
@elasticmachine

elasticmachine May 16, 2017

Collaborator

[golint] reported by reviewdog 🐶
exported method Loader.Generate should have comment or be unexported

@tsg

This comment has been minimized.

Copy link
Collaborator Author

commented May 16, 2017

@ruflin Yeah, I guess we should review the whole -setup mechanism, maybe we can unify the handling in such a way that it makes sense. The beats-tools is tempting, but we'll need to build it for all supported platforms, which makes me a bit worried about the packaging overhead (especially since it's not actually a beat).

I have the feeling we can't get a nice solution as part of this PR, so the question is what do we want for the time being? IMO stopping is much nicer in this case, but it's inconsistent with -setup, that's true :(. I'm still leaning towards stopping.

@ruflin

This comment has been minimized.

Copy link
Collaborator

commented May 16, 2017

Agree that we can't find a good solution as part of this PR. +1 on stopping as if you want to manually load the template, you don't want to already send data.

tsg added some commits May 17, 2017

@tsg tsg removed the in progress label May 17, 2017

@ruflin

ruflin approved these changes May 18, 2017

Copy link
Collaborator

left a comment

I left a few comments which we should handle in a follow up. Will merge it for now as this is a big step forward.

@@ -469,8 +469,30 @@ func (b *Beat) registerTemplateLoading() error {
}
}

esConfig := b.Config.Output["elasticsearch"]
// Check if outputting to file is enabled, and output to file if it is

This comment has been minimized.

Copy link
@ruflin

ruflin May 18, 2017

Collaborator

This comment confused me at first because the check for the file only happens later, but I see you meant it for the "complete" block

Settings templateSettings `config:"settings"`
OutputToFile OutputToFile `config:"output_to_file"`

This comment has been minimized.

Copy link
@ruflin

ruflin May 18, 2017

Collaborator

Not too much of a fan of the config name, but well :-)

@@ -80,6 +82,43 @@ func (l *Loader) Load() error {
return nil
}

// Generate generates the template and writes it to a file based on the configuration
// from `output_to_file`.
func (l *Loader) Generate() error {

This comment has been minimized.

Copy link
@ruflin

This comment has been minimized.

Copy link
@tsg

tsg May 23, 2017

Author Collaborator

Oh, I didn't realize we have that utility already.

@@ -9,4 +9,11 @@ def setUpClass(self):
self.beat_name = "mockbeat"
self.beat_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "../../"))
self.test_binary = self.beat_path + "/libbeat.test"
self.beats = [
"filebeat",

This comment has been minimized.

Copy link
@ruflin

ruflin May 18, 2017

Collaborator

Looks like a beats specifics dependency in libbeat.

This comment has been minimized.

Copy link
@tsg

tsg May 23, 2017

Author Collaborator

Yeah, only needed for those skipped tets for now, I can delete them.

beats = ["metricbeat", "packetbeat", "filebeat", "winlogbeat"]

for beat in beats:
for beat in self.beats:

This comment has been minimized.

Copy link
@ruflin

ruflin May 18, 2017

Collaborator

be aware that these tests are currently skipped.

@ruflin ruflin merged commit f9c0af6 into elastic:master May 18, 2017

6 checks passed

CLA Commit author is a member of Elasticsearch
Details
codecov/patch 37.14% of diff hit (within 100% threshold of 63.04%)
Details
codecov/project 63.41% (+0.36%) compared to b2ccdcb
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
default Build finished.
Details

ramon-garcia added a commit to ramon-garcia/beats that referenced this pull request Dec 5, 2017

Add the option to generate the template into a file (elastic#4323)
Part of elastic#3654.

This adds two settings: `setup.template.output_to_file.path` and `setup.template.output_to_file.version`. The version refers to the ES version and is optional, we'll use the Beats version if not specified. I put it under `output_to_file` to make it clear that it only applies when outputting to a file:

To generate a config, one can do:

```
./metricbeat -e -E setup.template.output_to_file.path=template.json
```

In the current version, the Beat automatically stops after generating the template, but the output might be slightly confusing:

```
2017/05/16 09:55:02.043671 load.go:116: INFO Template for Elasticsearch 6.0.0-alpha2 written to: template.json
2017/05/16 09:55:02.043717 beat.go:538: CRIT Exiting: Stopping after successfully writing the template to the file.
Exiting: Stopping after successfully writing the template to the file.
```

IMO this is better than the alternative of leaving it running.

To generate the template for the 2.x version, one can do:

```
./metricbeat -e -E setup.template.output_to_file.path=template.json -E setup.template.output_to_file.version=2.4.0
```

Remaining TODOs:

* [x] System test
* [x] Docs
* [x] Changelog

athom added a commit to athom/beats that referenced this pull request Jan 25, 2018

Add the option to generate the template into a file (elastic#4323)
Part of elastic#3654.

This adds two settings: `setup.template.output_to_file.path` and `setup.template.output_to_file.version`. The version refers to the ES version and is optional, we'll use the Beats version if not specified. I put it under `output_to_file` to make it clear that it only applies when outputting to a file:

To generate a config, one can do:

```
./metricbeat -e -E setup.template.output_to_file.path=template.json
```

In the current version, the Beat automatically stops after generating the template, but the output might be slightly confusing:

```
2017/05/16 09:55:02.043671 load.go:116: INFO Template for Elasticsearch 6.0.0-alpha2 written to: template.json
2017/05/16 09:55:02.043717 beat.go:538: CRIT Exiting: Stopping after successfully writing the template to the file.
Exiting: Stopping after successfully writing the template to the file.
```

IMO this is better than the alternative of leaving it running.

To generate the template for the 2.x version, one can do:

```
./metricbeat -e -E setup.template.output_to_file.path=template.json -E setup.template.output_to_file.version=2.4.0
```

Remaining TODOs:

* [x] System test
* [x] Docs
* [x] Changelog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.