Enable agent to send custom headers to kibana/ES #26275

michalpristas · 2021-06-14T09:24:11Z

What does this PR do?

This PR modifies container and enroll commands to accept list of headers which are then sent to kibana/ES when fleet-server or metric/file beat send events or performs setup.

Headers are hold in memory and are added to ES output in opt-in manner using inject_custom_headers rules in a spec file.

Why is it important?

To stay in par with other products working in cloud

Fixes: ##26137

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

…ader

elasticmachine · 2021-06-14T09:24:13Z

Pinging @elastic/agent (Team:Agent)

elasticmachine · 2021-06-14T09:27:29Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #26275 updated
Start Time: 2021-06-17T18:39:19.879+0000
Duration: 81 min 0 sec
Commit: 58f3526

Test stats 🧪

Test	Results
Failed	0
Passed	6924
Skipped	16
Total	6940

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	6924
Skipped	16
Total	6940

michalpristas · 2021-06-14T11:22:11Z

/package

michalpristas · 2021-06-14T15:10:32Z

/package

blakerouse · 2021-06-14T19:13:13Z

x-pack/elastic-agent/pkg/agent/application/info/agent_id.go

@@ -29,6 +29,7 @@ const maxRetriesloadAgentInfo = 5

 type persistentAgentInfo struct {
 	ID             string                                 `json:"id" yaml:"id" config:"id"`
+	CustomHeaders  map[string]string                      `json:"custom_headers" yaml:"custom_headers" config:"custom_headers"`


Why prefix with custom? Why not just "headers"?

That would match the elasticsearch configuration - https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#_headers

following naming from cloud

They are asking for the feature but we should align the naming with what is in beats to easy the transition to our users from beats to Elastic Agent. Unless we feel that headers is not a good name and now is the time to change it. I feel that headers is a better name over custom_headers.

x-pack/elastic-agent/pkg/agent/cmd/enroll.go

x-pack/elastic-agent/pkg/agent/cmd/setup_config.go

blakerouse · 2021-06-14T21:08:01Z

x-pack/elastic-agent/pkg/agent/cmd/setup_config.go

+			InsecureHTTP:        envBool("FLEET_SERVER_INSECURE_HTTP"),
+			PolicyID:            envWithDefault("", "FLEET_SERVER_POLICY_ID", "FLEET_SERVER_POLICY"),
+			Port:                envWithDefault("", "FLEET_SERVER_PORT"),
+			CustomHeaders:       envMap("FLEET_CUSTOM_HEADER"),


I would go with FLEET_HEADER instead of FLEET_CUSTOM_HEADER.

blakerouse · 2021-06-14T21:08:27Z

x-pack/elastic-agent/pkg/agent/transpiler/rules.go

@@ -90,6 +91,8 @@ func (r *RuleList) MarshalYAML() (interface{}, error) {
 			name = "fix_stream"
 		case *InsertDefaultsRule:
 			name = "insert_defaults"
+		case *InjectCustomHeadersRule:


How about just inject_headers?

x-pack/elastic-agent/pkg/agent/transpiler/rules.go

ph · 2021-06-15T14:42:23Z

@michalpristas Can you link the related issue in the description? I also moved it to the iteration board.

blakerouse · 2021-06-15T17:47:41Z

x-pack/elastic-agent/pkg/agent/cmd/enroll.go

@@ -59,6 +60,7 @@ func addEnrollFlags(cmd *cobra.Command) {
 	cmd.Flags().Uint16P("fleet-server-port", "", 0, "Fleet Server HTTP binding port (overrides the policy)")
 	cmd.Flags().StringP("fleet-server-cert", "", "", "Certificate to use for exposed Fleet Server HTTPS endpoint")
 	cmd.Flags().StringP("fleet-server-cert-key", "", "", "Private key to use for exposed Fleet Server HTTPS endpoint")
+	cmd.Flags().StringSliceP("header", "", []string{}, "App auth token used for elasticsearch")


Documentation for this flag is incorrect, seems like a copy/paste error.

thanks for noticing

cla-checker-service · 2021-06-17T06:31:00Z

💚 CLA has been signed

blakerouse

Thanks for the change to headers. Looks good!

…26276) * fixing some typos in the default oracle config, and adding default values * adding changelog entry

…lastic#26263) In the script processor, the `decode_xml` and `decode_xml_wineventlog` processors are now available as `DecodeXML` and `DecodeXMLWineventlog` respectively. This follows the same naming conventions as other processors exposed though the `script` processor. For example you can write `new processor.DecodeXML({...})`.

…stic#26273) * Add preserve_original_event option to o365audit input * Use String method from MapStr * Add test

Use configured CPU as total, use it to calculate free too. Total and free resources are only reported if they are configured with a total greater than zero.

* Add log.flags and object metadata to aws-s3 input events This adds the log.flags field created by the line readers to aws-s3 events. log.flags contains metadata like `multiline` and `truncated` to indicate how the data was processed. This also adds a config option to include S3 object metadata in the event if it exists. The use case for me was to get the Last-Modified timestamp for cases where the log does not have a timestamp or it cannot be parsed. Then this can be used as a fallback. * Pass metadata using s3Info struct to avoid adding new func params

The libbeat LineReader implementation did not handle the case where the underlying io.Reader it was reading from returns bytes and io.EOF. It was discarding the data in this case. As per the io.Reader contract: a Reader returning a non-zero number of bytes at the end of the input stream may return either err == EOF or err == nil. This occurs often with the gzip.Reader. It returns a large chunk of data at the end of the file and io.EOF at the same time.

…stic#25771) * initial commit * finux linux refactor * fix up freebsd * port main metricset, start openbsd * start work on openbsd vagrantfile * refactors of API, add darwin support * fix darwin implementation * refactor API, move tests, remove old code, take a crack at AIX * fix aix init func * fix tests * regenerate core data.json * small fixes, fix host field * update tests * run correct mage commands * try to fix system tests * more fixes for windows python tests * refactor CPU struct, use reflection * refactor reflection code, add validation * move metrics to its own internal folder * move directories, again * move directories, again * use optional Uint type * cleanup opt files * move around naming of opt types * fix up if block * change opt names * move around opt methods, cpu stat reader refactor * fix IsZero usage * add changelog

* Don't include full ES index template in errors Index templates in some beats can be very large (~1MB) and including the data in errors can use a lot of memory and also makes for very large log lines. If the error is recurring then this makes the effects worse. So this change removes the index template body from the error. Users that need to see the index template for debugging can use `<beatname> export template --es.version=1.2.3`. Fixes elastic#25540 * Update expected log messages in tests

…5999) * elastic#25990: Add `url.extension` to Packetbeat HTTP events * update changelog * add tests * updated per comment

…lastic#26324)

Updated filter expression for filtering 86 artifacts (elastic#26313)

* Forward port 7.13.2 changelog to 7.x (elastic#26315) * docs: Prepare Changelog for 7.13.2 (elastic#26302) * docs: Close changelog for 7.13.2 * Cleanup Co-authored-by: Andres Rodriguez <andresrc@gmail.com> (cherry picked from commit 0075f76) * Edit 7.13.2 changelog (elastic#26304) * Edit 7.13.2 changelog * Remove unneeded line Co-authored-by: Andres Rodriguez <andresrc@gmail.com> (cherry picked from commit 460ebf2) * Cleanup and fixes * More cleanup and fixes Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: DeDe Morton <dede.morton@elastic.co> (cherry picked from commit 3c9c96a) * Cleanup

MongoDB 4.4 uses structured JSON format for logging, add support for this format while keeping support for the old one too.

mergify · 2021-06-17T17:55:06Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feat/custom-header upstream/feat/custom-header
git merge upstream/master
git push upstream feat/custom-header

Enable agent to send custom headers to kibana/ES (elastic#26275)

…bana/ES (#26362) * update * Update CHANGELOG.next.asciidoc

Enable agent to send custom headers to kibana/ES (#26275)

* master: (25 commits) Fix UBI source URL (elastic#26384) Skip test_rotating_file in osx and windows (elastic#26379) Remove outdated k8s manifests for managed elastic-agent (elastic#26368) Enable agent to send custom headers to kibana/ES (elastic#26275) [Automation] Update elastic stack version to 8.0.0-943ef2c0 for testing (elastic#26354) Make the Syslog input GA (elastic#26293) Move Kerberos FAST config flag to shared kerberos config (elastic#26141) Add k8s cluster identifiers (elastic#26056) Store message from MongoDB json logs in message field (elastic#26338) update threatintel ECS version (elastic#26274) update envoyproxy ECS version (elastic#26277) [Filebeat] [MongoDB] Support MongoDB 4.4 json logs (elastic#24774) Update go-structform to 0.0.9 (elastic#26251) Forward port 7.13.2 changelog to master (elastic#26323) Updated filter expression for filtering 86 artifacts (elastic#26313) Osquerybeat: Align with the rest of the beats, set the ECS version (elastic#26324) [Packetbeat] Add `url.extension` to Packetbeat HTTP events (elastic#25999) Change link to snapshots in README (elastic#26317) Don't include full ES index template in errors (elastic#25743) First refactor of the system module - system/cpu and system/core (elastic#25771) ...

michalpristas added 3 commits June 10, 2021 14:28

add custom headers

becc30b

Merge branch 'master' of github.com:elastic/beats into feat/custom-he…

8d91386

…ader

headers

4515d5e

michalpristas added enhancement v7.14.0 Team:Elastic-Agent Label for the Agent team labels Jun 14, 2021

michalpristas self-assigned this Jun 14, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 14, 2021

changelog

de598b3

rename

9ebe416

michalpristas added 3 commits June 14, 2021 13:48

add kibana headers for setup

e92ef42

missing headers def

d9c0e99

missing headers def

b159ebd

blakerouse reviewed Jun 14, 2021

View reviewed changes

michalpristas added 2 commits June 15, 2021 12:55

headers

8695fed

rename

449476d

blakerouse reviewed Jun 15, 2021

View reviewed changes

Update enroll.go

029203b

blakerouse approved these changes Jun 17, 2021

View reviewed changes

P1llus and others added 4 commits June 17, 2021 19:53

[Oracle] Fixing default values for paths in config template (elastic#…

c711673

…26276) * fixing some typos in the default oracle config, and adding default values * adding changelog entry

[filebeat] Add preserve_original_event option to o365audit input (ela…

5f9e7a5

…stic#26273) * Add preserve_original_event option to o365audit input * Use String method from MapStr * Add test

Report total and free CPU for vSphere virtual machines (elastic#26167)

ef933b4

Use configured CPU as total, use it to calculate free too. Total and free resources are only reported if they are configured with a total greater than zero.

michel-laterman and others added 17 commits June 17, 2021 19:53

Update go version to 1.16.5 (elastic#26186)

2b053a3

Add support note (elastic#26283)

553be20

Disable packaging-arm in elastic-agent (elastic#26240)

da028ac

Change link to snapshots in README (elastic#26317)

ee2777d

[Packetbeat] Add url.extension to Packetbeat HTTP events (elastic#2…

2d8b720

…5999) * elastic#25990: Add `url.extension` to Packetbeat HTTP events * update changelog * add tests * updated per comment

Osquerybeat: Align with the rest of the beats, set the ECS version (e…

e5e74e3

…lastic#26324)

Updated filter expression for filtering 86 artifacts (elastic#26313)

5c5e7d8

Updated filter expression for filtering 86 artifacts (elastic#26313)

Update go-structform to 0.0.9 (elastic#26251)

e608bb6

[Filebeat] [MongoDB] Support MongoDB 4.4 json logs (elastic#24774)

939964b

MongoDB 4.4 uses structured JSON format for logging, add support for this format while keeping support for the old one too.

update envoyproxy ECS version (elastic#26277)

12a2fd0

update threatintel ECS version (elastic#26274)

255e83c

rename

b9ee11a

michalpristas force-pushed the feat/custom-header branch from 60128b9 to b9ee11a Compare June 17, 2021 17:54

michalpristas requested review from a team as code owners June 17, 2021 17:54

conflicts

58f3526

michalpristas merged commit 69d8ae7 into elastic:master Jun 18, 2021

michalpristas added a commit to michalpristas/beats that referenced this pull request Jun 18, 2021

Enable agent to send custom headers to kibana/ES (elastic#26275)

6e9b504

Enable agent to send custom headers to kibana/ES (elastic#26275)

This was referenced Jun 18, 2021

Cherry-pick #26275 to 7.x: Enable agent to send custom headers to kibana/ES #26361

Merged

Cherry-pick #26275 to 7.13: Enable agent to send custom headers to kibana/ES #26362

Merged

michalpristas added a commit that referenced this pull request Jun 18, 2021

Cherry-pick #26275 to 7.13: Enable agent to send custom headers to ki…

c9f0fc8

…bana/ES (#26362) * update * Update CHANGELOG.next.asciidoc

michalpristas added a commit that referenced this pull request Jun 18, 2021

Enable agent to send custom headers to kibana/ES (#26275) (#26361)

e4aa68a

Enable agent to send custom headers to kibana/ES (#26275)

cmacknz mentioned this pull request Sep 19, 2022

Agent V2: Automatically add agent info to Elasticsearch headers #32993

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable agent to send custom headers to kibana/ES #26275

Enable agent to send custom headers to kibana/ES #26275

michalpristas commented Jun 14, 2021 •

edited

elasticmachine commented Jun 14, 2021

elasticmachine commented Jun 14, 2021 •

edited by jenkins-beats-ci bot

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

michalpristas commented Jun 14, 2021

michalpristas commented Jun 14, 2021

blakerouse Jun 14, 2021

michalpristas Jun 15, 2021

blakerouse Jun 15, 2021

blakerouse Jun 14, 2021

blakerouse Jun 14, 2021

ph commented Jun 15, 2021

blakerouse Jun 15, 2021

michalpristas Jun 15, 2021

cla-checker-service bot commented Jun 17, 2021 •

edited

blakerouse left a comment

mergify bot commented Jun 17, 2021

Enable agent to send custom headers to kibana/ES #26275

Enable agent to send custom headers to kibana/ES #26275

Conversation

michalpristas commented Jun 14, 2021 • edited

What does this PR do?

Why is it important?

Checklist

elasticmachine commented Jun 14, 2021

elasticmachine commented Jun 14, 2021 • edited by jenkins-beats-ci bot

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

michalpristas commented Jun 14, 2021

michalpristas commented Jun 14, 2021

blakerouse Jun 14, 2021

Choose a reason for hiding this comment

michalpristas Jun 15, 2021

Choose a reason for hiding this comment

blakerouse Jun 15, 2021

Choose a reason for hiding this comment

blakerouse Jun 14, 2021

Choose a reason for hiding this comment

blakerouse Jun 14, 2021

Choose a reason for hiding this comment

ph commented Jun 15, 2021

blakerouse Jun 15, 2021

Choose a reason for hiding this comment

michalpristas Jun 15, 2021

Choose a reason for hiding this comment

cla-checker-service bot commented Jun 17, 2021 • edited

blakerouse left a comment

Choose a reason for hiding this comment

mergify bot commented Jun 17, 2021

michalpristas commented Jun 14, 2021 •

edited

elasticmachine commented Jun 14, 2021 •

edited by jenkins-beats-ci bot

cla-checker-service bot commented Jun 17, 2021 •

edited