Report FQDN on host.name as opt-in, enabled by feature flag (#2218)

* wip

* wip

* wip

* clean up

* wip

* .

* agent.name as fqdn

* it works

* wip

* WIP - using fake input, created test, it seems to work

* finish test for feature flags

* remove debug log

* some clean up

* mage check

* fixing licenses

* it works

* it works

* remove debug logs and comments

* make notice

* fake component and shipper are built by TestMain

* fixing tests

* make notice

* adjust tests

* update elatic-agent-client

* add .exe for win binaries

* adjust fake components build

* increase test timeout

* fix tests

* adjust features pkg and don't send featuresFlag if nil

* it works

* make notice

* revert some changes

* handle error properly

* add TODO

* Set FQDN for Vagrant elastic-agent VM

Useful for testing FQDN feature; harmless otherwise

* Undoing trivial formatting changes

* Running mage fmt

* Remove personal repo from NOTICE overrides

* Sorting imports

* Fix error handling

* Fixing access

* Fixing packaging of fake component

* Update logging

* Refactoring fake component

* Allowing fmt.Print* in TestMain

* Remove redundant return

* Fixing typo in error message

* Undo unintentional changes

* Removing redundant TestMain file

* Add comment on FQDN to Vagrantfile

* Remove debug logging statements

* Making test pass

* Document feature flags configuration in reference config file

* Adding CHANGELOG entry

* Use convenience method

* Formatting

* Use convenience method from elastic-agent-libs

* Running mage update

* Fixing imports

* Reducing whitespace changes

* Reducing whitespace changes

* Reducing whitespace changes

* Reducing whitespace changes

* Restore timeout duration

* Move mutex inside struct

* Prevent mutex copying

* Fixing state diagnostics test

* Fixing components diagnostics test

* Adding features.source to components golden file

* More interesting diagnostics state test

* Log error if FQDN lookup fails and fallback to OS-reported hostname

* Bump up go-sysinfo dependency version

* Fix typos

* Fixing compile errors

* Fix tests

* Fix imports

* Fix issues lost in rebase

* Fixing more imports

* Updating NOTICE.txt

* Clarify features indentation

* Remove replace directive

* Bumping up version on elastic-agent-client dependency

* Adding test mocking DNS

* Initialize featuresIdx in same place as units' configIdx

* Updating NOTICE.txt

* Remove unnecessary nil check

* Set features in state initially

* Update features in state from checkin observed message

* Remove TODO

* Updating elatic-agent.yml test fixture to enable FQDN feature flag

* Try flipping the authoritative flag

* Remove unreliable DNS mocking test

* Set source

* Update test fixture

* Change logger

* Fixing test case

* Omit feature flags serialization from state.yaml diag file

* Updating NOTICE.txt

* Add descriptive comment on why Features is not being serialized as YAML

* Updating test fixture

* Inject logger into metadata methods

---------

Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com>
(cherry picked from commit be818cc)

Vagrantfile

@@ -131,7 +131,10 @@ Vagrant.configure("2") do |config|
 
     config.vm.define "elastic-agent" do |nodeconfig|
       nodeconfig.vm.box = "ubuntu/jammy64"
-      nodeconfig.vm.hostname = "elastic-agent-dev"
+
+      # We deliberately set a fully-qualified domain name for the VM; it helps
+      # test the FQDN feature flag.
+      nodeconfig.vm.hostname = "elastic-agent-dev.elastic.dev.internal"
 
       nodeconfig.vm.network "private_network",
         hostname: true,

_meta/config/common.reference.p2.yml.tmpl

@@ -154,6 +154,13 @@ inputs:
 #   # period define how frequent we should look for changes in the configuration.
 #   period: 10s
 
+# Feature Flags
+
+# This section enables or disables feature flags supported by Agent and its components.
+#agent.features:
+#  fqdn:
+#    enabled: false
+
 # Logging
 
 # There are four options for the log output: file, stderr, syslog, eventlog

changelog/fragments/1679422558-feature-flags.yaml

@@ -0,0 +1,34 @@
+# Kind can be one of:
+# - breaking-change: a change to previously-documented behavior
+# - deprecation: functionality that is being removed in a later release
+# - bug-fix: fixes a problem in a previous version
+# - enhancement: extends functionality but does not break or fix existing behavior
+# - feature: new functionality
+# - known-issue: problems that we are aware of in a given version
+# - security: impacts on the security of a product or a user’s deployment.
+# - upgrade: important information for someone upgrading from a prior version
+# - other: does not fit into any of the other categories
+kind: feature
+
+# Change summary; a 80ish characters long description of the change.
+summary: feature-flags
+
+# Long description; in case the summary is not enough to describe the change
+# this field accommodate a description without length limits.
+# NOTE: This field will be rendered only for breaking-change and known-issue kinds at the moment.
+description: |
+  Add support for feature flags, starting with one to toggle FQDN reporting in
+  events generated by Agent components, via the `host.name` field.
+
+# Affected component; a word indicating the component this changeset affects.
+component: agent
+
+# PR URL; optional; the PR number that added the changeset.
+# If not present is automatically filled by the tooling finding the PR where this changelog fragment has been added.
+# NOTE: the tooling supports backports, so it's able to fill the original PR number instead of the backport PR number.
+# Please provide it if you are adding a fragment for a different PR.
+pr: https://github.com/elastic/elastic-agent/pull/2218
+
+# Issue URL; optional; the GitHub issue related to this changeset (either closes or is part of).
+# If not present is automatically filled by the tooling with the issue linked to the PR number.
+issue: https://github.com/elastic/elastic-agent/issues/2185

elastic-agent.reference.yml

@@ -160,6 +160,13 @@ inputs:
 #   # period define how frequent we should look for changes in the configuration.
 #   period: 10s
 
+# Feature Flags
+
+# This section enables or disables feature flags supported by Agent and its components.
+#agent.features:
+#  fqdn:
+#    enabled: false
+
 # Logging
 
 # There are four options for the log output: file, stderr, syslog, eventlog

go.mod

@@ -3,25 +3,25 @@ module github.com/elastic/elastic-agent
 go 1.19
 
 require (
-	github.com/Microsoft/go-winio v0.5.2
+	github.com/Microsoft/go-winio v0.6.0
 	github.com/antlr/antlr4 v0.0.0-20200820155224-be881fa6b91d
 	github.com/billgraziano/dpapi v0.4.0
 	github.com/blakesmith/ar v0.0.0-20150311145944-8bd4349a67f2
 	github.com/cavaliercoder/go-rpm v0.0.0-20190131055624-7a9c54e3d83e
 	github.com/coreos/go-systemd/v22 v22.3.3-0.20220203105225-a9a7ef127534
-	github.com/docker/go-units v0.4.0
+	github.com/docker/go-units v0.5.0
 	github.com/dolmen-go/contextio v0.0.0-20200217195037-68fc5150bcd5
 	github.com/elastic/e2e-testing v1.99.2-0.20221205111528-ade3c840d0c0
 	github.com/elastic/elastic-agent-autodiscover v0.2.1
-	github.com/elastic/elastic-agent-client/v7 v7.0.2-0.20221129150247-15881a8e64ef
+	github.com/elastic/elastic-agent-client/v7 v7.0.3-0.20230315204017-166fd1fd746f
 	github.com/elastic/elastic-agent-libs v0.3.6
-	github.com/elastic/elastic-agent-system-metrics v0.4.4
+	github.com/elastic/elastic-agent-system-metrics v0.4.6-0.20230308003052-ba171438211e
 	github.com/elastic/go-licenser v0.4.0
-	github.com/elastic/go-sysinfo v1.8.1
+	github.com/elastic/go-sysinfo v1.9.1-0.20230328042007-6dcfe88b8359
 	github.com/elastic/go-ucfg v0.8.6
 	github.com/gofrs/flock v0.8.1
 	github.com/gofrs/uuid v4.2.0+incompatible
-	github.com/google/go-cmp v0.5.6
+	github.com/google/go-cmp v0.5.9
 	github.com/gorilla/mux v1.8.0
 	github.com/hashicorp/go-multierror v1.1.1
 	github.com/hectane/go-acl v0.0.0-20190604041725-da78bae5fc95
@@ -38,7 +38,7 @@ require (
 	github.com/pkg/errors v0.9.1
 	github.com/rs/zerolog v1.27.0
 	github.com/shirou/gopsutil/v3 v3.21.12
-	github.com/sirupsen/logrus v1.8.1
+	github.com/sirupsen/logrus v1.9.0
 	github.com/spf13/cobra v1.3.0
 	github.com/stretchr/testify v1.8.2
 	github.com/tsg/go-daemon v0.0.0-20200207173439-e704b93fd89b
@@ -48,8 +48,8 @@ require (
 	go.uber.org/zap v1.21.0
 	golang.org/x/crypto v0.5.0
 	golang.org/x/lint v0.0.0-20210508222113-6edffad5e616
-	golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
-	golang.org/x/sys v0.4.0
+	golang.org/x/sync v0.1.0
+	golang.org/x/sys v0.6.0
 	golang.org/x/text v0.7.0
 	golang.org/x/time v0.3.0
 	golang.org/x/tools v0.1.12
@@ -69,12 +69,11 @@ require (
 	github.com/armon/go-radix v1.0.0 // indirect
 	github.com/cavaliercoder/badio v0.0.0-20160213150051-ce5280129e9e // indirect
 	github.com/cenkalti/backoff/v4 v4.1.2 // indirect
-	github.com/containerd/containerd v1.5.13 // indirect
 	github.com/cyphar/filepath-securejoin v0.2.3 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/dnephin/pflag v1.0.7 // indirect
 	github.com/docker/distribution v2.8.1+incompatible // indirect
-	github.com/docker/docker v20.10.12+incompatible // indirect
+	github.com/docker/docker v20.10.22+incompatible // indirect
 	github.com/docker/go-connections v0.4.0 // indirect
 	github.com/elastic/go-structform v0.0.9 // indirect
 	github.com/elastic/go-windows v1.0.1 // indirect
@@ -111,9 +110,10 @@ require (
 	github.com/opencontainers/image-spec v1.0.2 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
 	github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
-	github.com/prometheus/procfs v0.7.3 // indirect
+	github.com/prometheus/procfs v0.9.0 // indirect
 	github.com/santhosh-tekuri/jsonschema v1.2.4 // indirect
 	github.com/sergi/go-diff v1.1.0 // indirect
+	github.com/shirou/gopsutil v3.21.11+incompatible // indirect
 	github.com/spf13/pflag v1.0.5 // indirect
 	github.com/stretchr/objx v0.5.0 // indirect
 	github.com/tklauser/go-sysconf v0.3.9 // indirect
@@ -129,7 +129,6 @@ require (
 	golang.org/x/net v0.5.0 // indirect
 	golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8 // indirect
 	golang.org/x/term v0.4.0 // indirect
-	golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
 	google.golang.org/appengine v1.6.7 // indirect
 	google.golang.org/genproto v0.0.0-20220426171045-31bebdecfb46 // indirect
 	google.golang.org/grpc/examples v0.0.0-20220304170021-431ea809a767 // indirect