allow data stream namespace configuration for standalone

[graphaelli]

using apm-server.data_streams.namespace

Motivation/summary

closes #7313

Checklist

• Update CHANGELOG.asciidoc
• Documentation has been updated

How to test these changes

Start an unmanaged apm-server with apm-server.data_streams.namespace set and inspect the trace data streams created.

[mergify]

This pull request does not have a backport label. Could you fix it @graphaelli? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-7.x is the label to automatically backport to the 7.x branch.
• backport-7./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[apmmachine]

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-02-18T02:38:25.537+0000

• Duration: 57 min 6 sec

Test stats 🧪

Test	Results
Failed	1
Passed	5643
Skipped	19
Total	5663

Test errors

Expand to view the tests failures

Build and Test / System and Environment Tests / TestFleetIntegrationBeatsMonitoring – github.com/elastic/apm-server/systemtest

Expand to view the error details

 Failed 

Expand to view the stacktrace

 === RUN   TestFleetIntegrationBeatsMonitoring
2022/02/18 03:03:22 Building image elastic-agent-systemtest:8.2.0-74785f8b-SNAPSHOT...
2022/02/18 03:03:23 Built image elastic-agent-systemtest:8.2.0-74785f8b-SNAPSHOT
2022/02/18 03:03:23 Starting container id: 3ec359274373 image: elastic-agent-systemtest:8.2.0-74785f8b-SNAPSHOT
2022/02/18 03:03:24 Waiting for container id 3ec359274373 image: elastic-agent-systemtest:8.2.0-74785f8b-SNAPSHOT
2022/02/18 03:03:29 Container is ready id: 3ec359274373 image: elastic-agent-systemtest:8.2.0-74785f8b-SNAPSHOT
    fleet_test.go:78: 
        	Error Trace:	fleet_test.go:78
        	Error:      	Not equal: 
        	            	expected: map[string]interface {}{"output":map[string]interface {}{"events":map[string]interface {}{"acked":15, "active":0, "batches":1, "failed":0, "toomany":0, "total":15}}}
        	            	actual  : map[string]interface {}{"config":map[string]interface {}{"module":map[string]interface {}{"running":0, "starts":0, "stops":0}, "reloads":0, "scans":0}, "output":map[string]interface {}{"events":map[string]interface {}{"acked":0, "active":0, "batches":0, "dropped":0, "duplicates":0, "failed":0, "toomany":0, "total":0}, "read":map[string]interface {}{"bytes":0, "errors":0}, "type":"elasticsearch", "write":map[string]interface {}{"bytes":0, "errors":0}}, "pipeline":map[string]interface {}{"clients":0, "events":map[string]interface {}{"active":0, "dropped":0, "failed":0, "filtered":0, "published":0, "retry":0, "total":0}, "queue":map[string]interface {}{"acked":0, "max_events":4096}}}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,10 +1,46 @@
        	            	-(map[string]interface {}) (len=1) {
        	            	- (string) (len=6) "output": (map[string]interface {}) (len=1) {
        	            	-  (string) (len=6) "events": (map[string]interface {}) (len=6) {
        	            	-   (string) (len=5) "acked": (float64) 15,
        	            	+(map[string]interface {}) (len=3) {
        	            	+ (string) (len=6) "config": (map[string]interface {}) (len=3) {
        	            	+  (string) (len=6) "module": (map[string]interface {}) (len=3) {
        	            	+   (string) (len=7) "running": (float64) 0,
        	            	+   (string) (len=6) "starts": (float64) 0,
        	            	+   (string) (len=5) "stops": (float64) 0
        	            	+  },
        	            	+  (string) (len=7) "reloads": (float64) 0,
        	            	+  (string) (len=5) "scans": (float64) 0
        	            	+ },
        	            	+ (string) (len=6) "output": (map[string]interface {}) (len=4) {
        	            	+  (string) (len=6) "events": (map[string]interface {}) (len=8) {
        	            	+   (string) (len=5) "acked": (float64) 0,
        	            	    (string) (len=6) "active": (float64) 0,
        	            	-   (string) (len=7) "batches": (float64) 1,
        	            	+   (string) (len=7) "batches": (float64) 0,
        	            	+   (string) (len=7) "dropped": (float64) 0,
        	            	+   (string) (len=10) "duplicates": (float64) 0,
        	            	    (string) (len=6) "failed": (float64) 0,
        	            	    (string) (len=7) "toomany": (float64) 0,
        	            	-   (string) (len=5) "total": (float64) 15
        	            	+   (string) (len=5) "total": (float64) 0
        	            	+  },
        	            	+  (string) (len=4) "read": (map[string]interface {}) (len=2) {
        	            	+   (string) (len=5) "bytes": (float64) 0,
        	            	+   (string) (len=6) "errors": (float64) 0
        	            	+  },
        	            	+  (string) (len=4) "type": (string) (len=13) "elasticsearch",
        	            	+  (string) (len=5) "write": (map[string]interface {}) (len=2) {
        	            	+   (string) (len=5) "bytes": (float64) 0,
        	            	+   (string) (len=6) "errors": (float64) 0
        	            	+  }
        	            	+ },
        	            	+ (string) (len=8) "pipeline": (map[string]interface {}) (len=3) {
        	            	+  (string) (len=7) "clients": (float64) 0,
        	            	+  (string) (len=6) "events": (map[string]interface {}) (len=7) {
        	            	+   (string) (len=6) "active": (float64) 0,
        	            	+   (string) (len=7) "dropped": (float64) 0,
        	            	+   (string) (len=6) "failed": (float64) 0,
        	            	+   (string) (len=8) "filtered": (float64) 0,
        	            	+   (string) (len=9) "published": (float64) 0,
        	            	+   (string) (len=5) "retry": (float64) 0,
        	            	+   (string) (len=5) "total": (float64) 0
        	            	+  },
        	            	+  (string) (len=5) "queue": (map[string]interface {}) (len=2) {
        	            	+   (string) (len=5) "acked": (float64) 0,
        	            	+   (string) (len=10) "max_events": (float64) 4096
        	            	   }
        	Test:       	TestFleetIntegrationBeatsMonitoring
    fleet_test.go:151: elastic-agent logs: {"log.level":"info","@timestamp":"2022-02-18T03:03:24.825Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":455},"message":"Starting enrollment to URL: https://fleet-server:8220/","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-02-18T03:03:25.598Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":253},"message":"Elastic Agent might not be running; unable to trigger restart","ecs.version":"1.6.0"}
        Successfully enrolled the Elastic Agent.
        {"log.level":"info","@timestamp":"2022-02-18T03:03:25.707Z","log.origin":{"file.name":"application/application.go","file.line":68},"message":"Detecting execution mode","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-02-18T03:03:25.708Z","log.origin":{"file.name":"application/application.go","file.line":92},"message":"Agent is managed by Fleet","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-02-18T03:03:25.708Z","log.origin":{"file.name":"capabilities/capabilities.go","file.line":59},"message":"capabilities file not found in /usr/share/elastic-agent/state/capabilities.yml","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-02-18T03:03:26.531Z","log.logger":"composable.providers.docker","log.origin":{"file.name":"docker/docker.go","file.line":43},"message":"Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-02-18T03:03:26.533Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":63},"message":"Starting stats endpoint","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-02-18T03:03:26.533Z","log.origin":{"file.name":"application/managed_mode.go","file.line":290},"message":"Agent is starting","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-02-18T03:03:26.533Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":65},"message":"Metrics endpoint listening on: /usr/share/elastic-agent/state/data/tmp/elastic-agent.sock (configured: unix:///usr/share/elastic-agent/state/data/tmp/elastic-agent.sock)","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-02-18T03:03:28.096Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"New State ID is PfLsKboh","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-02-18T03:03:28.097Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":49},"message":"Converging state requires execution of 2 step(s)","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-02-18T03:03:28.367Z","log.origin":{"file.name":"operation/operator.go","file.line":301},"message":"operation 'operation-install' skipped for apm-server.8.2.0-SNAPSHOT","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-02-18T03:03:28.926Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-02-18T03:03:28Z - message: Application: apm-server--8.2.0-SNAPSHOT[4ce01d7b-2642-4c80-b895-e718f88bf847]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
        {"log.level":"info","@timestamp":"2022-02-18T03:03:29.989Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-02-18T03:03:29Z - message: Application: apm-server--8.2.0-SNAPSHOT[4ce01d7b-2642-4c80-b895-e718f88bf847]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
--- FAIL: TestFleetIntegrationBeatsMonitoring (19.02s)
 

Steps errors

Expand to view the steps failures

Run Linux tests

• Took 14 min 56 sec . View more details here
• Description: ./.ci/scripts/linux-test.sh

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /hey-apm : Run the hey-apm benchmark.

• /package : Generate and publish the docker images.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[axw]

LGTM, thanks!

[graphaelli]

we'll need #7315 merged in before this will pass CI

[mergify]

This pull request is now in conflicts. Could you fix it @graphaelli? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b standalone-ds-namespace upstream/standalone-ds-namespace
git merge upstream/main
git push upstream standalone-ds-namespace