[Ingest Manager] Expose processes and their metrics #24788

michalpristas · 2021-03-26T12:55:07Z

What does this PR do?

Added /processes and /processes/{processID} endpoints to http server.

agent has its server on unix:///tmp/elastic-agent/elastic-agent.sock or npipe:///elastic-agent for windows.
not configurable

example of /processes

{
	"processes": [{
		"id": "filebeat-default-monitoring",
		"pid": "8025",
		"binary": "filebeat",
		"source": {
			"kind": "internal",
			"outputs": ["default"]
		}
	}, {
		"id": "metricbeat-default-monitoring",
		"pid": "8043",
		"binary": "metricbeat",
		"source": {
			"kind": "internal",
			"outputs": ["default"]
		}
	}, {
		"id": "metricbeat-default",
		"pid": "7998",
		"binary": "metricbeat",
		"source": {
			"kind": "configured",
			"outputs": ["default"]
		}
	}]
}

example of /processes/metricbeat-default

{
	"beat": {
		"cgroup": {
			"cpu": {
				"cfs": {
					"period": {
						"us": 100000
					},
					"quota": {
						"us": 0
					}
				},
				"id": "user.slice",
				"stats": {
					"periods": 0,
					"throttled": {
						"ns": 0,
						"periods": 0
					}
				}
			},
			"cpuacct": {
				"id": "user.slice",
				"total": {
					"ns": 994162833024
				}
			},
			"memory": {
				"id": "user.slice",
				"mem": {
					"limit": {
						"bytes": 9223372036854771712
					},
					"usage": {
						"bytes": 1766760448
					}
				}
			}
		},
		"cpu": {
			"system": {
				"ticks": 150,
				"time": {
					"ms": 156
				}
			},
			"total": {
				"ticks": 220,
				"time": {
					"ms": 232
				},
				"value": 220
			},
			"user": {
				"ticks": 70,
				"time": {
					"ms": 76
				}
			}
		},
		"handles": {
			"limit": {
				"hard": 1048576,
				"soft": 1024
			},
			"open": 17
		},
		"info": {
			"ephemeral_id": "aad52edf-4229-4927-bb30-c67ce9934499",
			"uptime": {
				"ms": 26728
			}
		},
		"memstats": {
			"gc_next": 16893216,
			"memory_alloc": 14370808,
			"memory_sys": 75056128,
			"memory_total": 34212120,
			"rss": 85266432
		},
		"runtime": {
			"goroutines": 58
		}
	},
	"libbeat": {
		"config": {
			"module": {
				"running": 4,
				"starts": 4,
				"stops": 0
			},
			"reloads": 1,
			"scans": 1
		},
		"output": {
			"events": {
				"acked": 0,
				"active": 0,
				"batches": 0,
				"dropped": 0,
				"duplicates": 0,
				"failed": 0,
				"toomany": 0,
				"total": 0
			},
			"read": {
				"bytes": 0,
				"errors": 0
			},
			"type": "elasticsearch",
			"write": {
				"bytes": 0,
				"errors": 0
			}
		},
		"pipeline": {
			"clients": 4,
			"events": {
				"active": 35,
				"dropped": 0,
				"failed": 0,
				"filtered": 0,
				"published": 35,
				"retry": 44,
				"total": 35
			},
			"queue": {
				"acked": 0
			}
		}
	},
	"metricbeat": {
		"system": {
			"cpu": {
				"events": 3,
				"failures": 0,
				"success": 3
			},
			"filesystem": {
				"events": 12,
				"failures": 0,
				"success": 12
			},
			"memory": {
				"events": 3,
				"failures": 0,
				"success": 3
			},
			"network": {
				"events": 17,
				"failures": 0,
				"success": 17
			}
		}
	},
	"system": {
		"cpu": {
			"cores": 4
		},
		"load": {
			"1": 0.58,
			"15": 1.43,
			"5": 1.43,
			"norm": {
				"1": 0.145,
				"15": 0.3575,
				"5": 0.3575
			}
		}
	}
}

in case of error e.g

{
	"type": "UNEXPECTED",
	"reason": "failed fetching metrics: Get \"http://unix/stats\": dial unix /tmp/elastic-agent/default/metricbeat/metricbeat.sock: connect: no such file or directory"
}

Why is it important?

Fixes: #24091

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

elasticmachine · 2021-03-26T13:15:19Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Started by user Michal Pristas
Start Time: 2021-04-08T18:09:50.933+0000
Duration: 146 min 15 sec
Commit: f83d7b7

Test stats 🧪

Test	Results
Failed	0
Passed	46934
Skipped	5132
Total	52066

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	46934
Skipped	5132
Total	52066

simitt

Thanks @michalpristas ! Checked out your PR and it generally works as described. Left some minor comments.

The processes endpoints need to be exposed via a TCP port though, so that the information can be queried from other containers via http request. The port needs to be configurable. It's fine to do this in a follow-up PR, but it is a requirement for cloud to be able to collect the information (required for 7.13).

simitt · 2021-03-29T08:19:02Z

x-pack/elastic-agent/pkg/core/monitoring/server/process.go

+)
+
+const (
+	procuctIDKey     = "processID"


The PR uses product, program and process at different places for logic dealing with processes. For consistency reasons, I think we should aim for always using process, which reduces the mental overhead.

simitt · 2021-03-29T08:23:56Z

x-pack/elastic-agent/pkg/core/monitoring/server/process.go

+		metricsBytes, metricsErr := processMetrics(r.Context(), id)
+		if metricsErr != nil {
+			resp := errResponse{
+				Type:   "UNEXPECTED",


used a couple of times, maybe worth introducing a typeUnexpected

michalpristas · 2021-03-29T09:20:10Z

i will make it exposed using configurable TCP in this PR, this is still a draft made in a way so you can pick it up as soon as possible.

@simitt is it ok to make it configurable in a way that when configuration is missing it wont be exposed so i dont use ports when i dont need to? is cloud capable of updating this port before agent start? or do you need static port from the start

ruflin · 2021-03-30T07:24:12Z

+1 on having it disabled by the default. Will this be configurable through Fleet or not? The current preference would be that the person running Elastic Agent can set this and not necessarily available in Fleet.

michalpristas · 2021-03-31T07:49:47Z

as is now it's not configurable from fleet

michalpristas · 2021-03-31T07:59:19Z

@simitt updated solution with HTTP endpoint, processes wont be exposed unless Port is specified in agent.monitoring.port config option.

As we did not received any feedback from cloud just yet, only port can be specified and it is exposed without SSL.

simitt · 2021-03-31T08:12:38Z

thanks @michalpristas; I'll give it a try as soon as possible

ruflin · 2021-03-31T08:20:37Z

I set this up on port 81.

For metricbeat monitoring and non monitoring, I get the info I was looking for

For http://localhost:81/processes/filebeat-default-monitoring I got

{"type":"UNEXPECTED","reason":"failed fetching metrics: Get \"http://unix/stats\": dial unix /tmp/elastic-agent/default/filebeat/filebeat.sock: connect: no such file or directory"}

I was a bit surprised that / returns a 404. Maybe we could add some basic info here?

Every time the /processes endpoint is reloaded, the order of the output changes. Not a deal breaker but it confused me at first as I thought the content changed.

ruflin · 2021-03-31T08:22:04Z

x-pack/elastic-agent/_meta/config/common.p2.yml.tmpl

+#   # process stats are exposed only using this option
+#   # it is up to a caller to make sure port is usable and free to use.
+#   # by default 0 is used meaning socket is used instead.
+#   port: 0


I suggest we use the same config blocks as we have in beats: https://github.com/elastic/beats/blob/master/filebeat/filebeat.reference.yml#L2507

We need a host to decide if it should only be exposed on localhost or broader. This also allows to add the enabled option.

Right, at least for when it is run in docker localhost would not be sufficient.

updated with http.enabled/host/port options

Great. @simitt I assume on Cloud we can just use this in the template and make the port configurable or hardcode.

elasticmachine · 2021-03-31T09:40:21Z

Pinging @elastic/agent (Team:Agent)

ruflin · 2021-04-06T06:22:16Z

x-pack/elastic-agent/_meta/config/elastic-agent.docker.yml.tmpl

+#       # When using IP addresses, it is recommended to only use localhost.
+#       host: localhost
+#       # Port on which the HTTP endpoint will bind. Default is 0 meaning feature is disabled.
+#       port: 0


As we have enabled / disabled now, do we still need the support for 0 ?

ruflin · 2021-04-06T11:29:09Z

x-pack/elastic-agent/pkg/core/monitoring/server/process.go

+			return
+		}
+
+		fmt.Fprint(w, string(metricsBytes))


I think I normally like is that metrics endpoint are also human readable. What I mean in the context here is that we pretty print the json. Unfortunately this means in this context to convert it first to json to be able to pretty print it with indentation. At the same time, should not cause too much overhead?

how do you specify pretty print with metricbeat? we can pass argument to mb if passed to agent

As far as I remember there is a ?pretty flag: /stats/?pretty. Not sure if that works over the socket. I was initially thinking to implement it here so it works for all the outputs also agent, but not strong preference.

I think the pretty was a special flag provided by the expvar handler. Not sure we have had it implemented in Beats.

BTW this is not a blocker, please ignore it for now.

ruflin · 2021-04-06T11:31:11Z

@michalpristas How do I reach the metric data from elastic-agent itself?

michalpristas · 2021-04-06T11:31:42Z

/stats endpoint as with beats

urso · 2021-04-06T20:12:50Z

x-pack/elastic-agent/pkg/core/monitoring/config/config.go

+type MonitoringHTTPConfig struct {
+	Enabled bool   `yaml:"enabled" config:"enabled"`
+	Host    string `yaml:"host" config:"host"`
+	Port    int    `yaml:"port" config:"port"`


Let's add a 'positive' validator (see unpack) docs. Then if port is configured, but empty we will fail to parse the configuration and fail with the setting that failed. The Enabled will be used to not start the server.

x-pack/elastic-agent/pkg/core/monitoring/server/process.go

urso · 2021-04-06T20:21:35Z

x-pack/elastic-agent/pkg/core/monitoring/server/process.go

+func processMetrics(ctx context.Context, id string) ([]byte, int, error) {
+	detail, err := parseID(id)
+	if err != nil {
+		return nil, http.StatusInternalServerError, err


This is no internal error, but the user did provide invalid input.

urso · 2021-04-06T20:22:23Z

x-pack/elastic-agent/pkg/core/monitoring/server/process.go

+			return
+		}
+
+		fmt.Fprint(w, string(metricsBytes))


I think the pretty was a special flag provided by the expvar handler. Not sure we have had it implemented in Beats.

x-pack/elastic-agent/pkg/core/monitoring/server/process.go

) * Add status sub-command to report status of running daemon. * Set exit code based on health status. * Add changelog. * Fix format.

Length limit is 104 on unix.

Sync legacy apm ingest folder to HOME dir.

* Add baseline ECS 1.9.0 upgrade * update changelog

* feat: stage execution cache * fix: use correct context * fix: do not check stage status on the first run * fix: proper URL * chore: show message when the stache is skip * fix: correct path * fix: add final / * test: is the path needed? * fix: remove prefix * chore: refactor to use curl to download * chore: use pipeline step

…c#24904) * Add check for URL set when cert and cert key. * Add changelog.

…o expose-processes

urso · 2021-04-07T20:48:51Z

x-pack/elastic-agent/pkg/core/monitoring/server/process.go

+		return nil, 0, errorWithStatus(http.StatusInternalServerError, err)
+	}
+
+	return rb, resp.StatusCode, nil


Can StatusCode be != 200 here?

i dont know, but i would rather proxy whatever is retrieved from beat than mix 200 with error message

[Ingest Manager] Expose processes and their metrics (elastic#24788)

Cherry-pick #24788 to 7.x: Expose processes and their metrics (#25017)

* upstream/master: (308 commits) [winlogbeat] Add support for sysmon v13 events 24 and 25 (elastic#24945) mergify: add backport label (elastic#25050) Add pod.ip in k8s metadata (elastic#25037) [elastic-agent] Use fleet.url for container cmd (elastic#25026) disable TestXPackEnabled flaky test in logstash metricbeat module (elastic#25034) Leverege leader election in agent k8s manifests (elastic#25016) libbeat/publisher/pipeline: expand monitoring (elastic#24700) libbeat: fix decode_json_fields config validation (elastic#24862) Remove make docs-preview instructions (elastic#25001) [Filebeat] Fix IPtables pipeline (elastic#24928) [DOCS] cd into correct directory before invoking mage. (elastic#17679) Add -buildmode=pie for supported platform (elastic#24964) Add agent's direcotry in k8s manifest generator (elastic#24987) [mergify] assign the original author (elastic#25007) Fix AWS module flaky tests (elastic#24852) [filebeat] Use fail_on_template_error on google_workspace and okta pagination (elastic#24967) Updated config to match defaults (elastic#25004) [Filebeat] Fix hardcoded amazonaws.com endpoint (elastic#24861) Add cloud.service.name to add_cloud_metadata (elastic#24993) [Ingest Manager] Expose processes and their metrics (elastic#24788) ...

michalpristas added 2 commits March 26, 2021 13:50

processes exposed

b9da49c

Merge branch 'master' of github.com:elastic/beats into expose-processes

cb72c8d

michalpristas added enhancement needs_backport PR is waiting to be backported to other branches. Team:Ingest Management Team:Elastic-Agent Label for the Agent team labels Mar 26, 2021

michalpristas requested a review from simitt March 26, 2021 12:55

michalpristas self-assigned this Mar 26, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Mar 26, 2021

michalpristas mentioned this pull request Mar 26, 2021

Exposing Data from Agent and APM Server #24091

Closed

michalpristas added 2 commits March 26, 2021 15:12

lint

1beb18c

notice

03a58cb

simitt reviewed Mar 29, 2021

View reviewed changes

michalpristas added 2 commits March 29, 2021 12:16

Merge branch 'master' of github.com:elastic/beats into expose-processes

f89a02e

http

482a141

disabled by default

cfce22d

config update

3798b74

ruflin reviewed Mar 31, 2021

View reviewed changes

michalpristas added 2 commits March 31, 2021 10:54

fixed tests

2d3ee37

allow specify enabled/host/port

07e3ff1

michalpristas marked this pull request as ready for review March 31, 2021 09:40

michalpristas added 2 commits April 1, 2021 16:18

Merge branch 'master' of github.com:elastic/beats into expose-processes

ed2b1f8

Merge branch 'master' of github.com:elastic/beats into expose-processes

74504cc

ruflin reviewed Apr 6, 2021

View reviewed changes

urso reviewed Apr 6, 2021

View reviewed changes

blakerouse and others added 14 commits April 7, 2021 08:37

Add status subcommand to report status of running daemon. (elastic#24856

af36c9a

) * Add status sub-command to report status of running daemon. * Set exit code based on health status. * Add changelog. * Fix format.

Add ability to destroy indices with wildcards in testing (elastic#24915)

965db5e

[Elastic Agent]: Reduce allowed socket path length (elastic#24914)

5677082

Length limit is 104 on unix.

[Elastic Agent] Cloud container legacy apm files. (elastic#24896)

d763189

Sync legacy apm ingest folder to HOME dir.

Add baseline ECS 1.9.0 upgrade (elastic#24909)

46ab666

* Add baseline ECS 1.9.0 upgrade * update changelog

Fix error in Journalbeat commands (elastic#24880)

d85b239

[Elastic Agent] Add check for URL set when cert and cert key. (elasti…

556dc82

…c#24904) * Add check for URL set when cert and cert key. * Add changelog.

[libbeat] Add wineventlog schema to decode_xml processor (elastic#24726)

2873c4f

status related to error

085b63c

Merge branch 'expose-processes' of github.com:michalpristas/beats int…

8730c4c

…o expose-processes

Merge branch 'master' of github.com:elastic/beats into expose-processes

6fbf8a9

lint

4bdca73

test

6e190ae

urso reviewed Apr 7, 2021

View reviewed changes

Merge branch 'master' of github.com:elastic/beats into expose-processes

f83d7b7

urso approved these changes Apr 8, 2021

View reviewed changes

michalpristas merged commit 9625db6 into elastic:master Apr 9, 2021

michalpristas mentioned this pull request Apr 9, 2021

[Ingest Manager] Updated config to match defaults #25004

Merged

6 tasks

michalpristas added a commit to michalpristas/beats that referenced this pull request Apr 12, 2021

[Ingest Manager] Expose processes and their metrics (elastic#24788)

4d8701f

[Ingest Manager] Expose processes and their metrics (elastic#24788)

michalpristas mentioned this pull request Apr 12, 2021

Cherry-pick #24788 to 7.x: Expose processes and their metrics #25017

Merged

6 tasks

michalpristas added a commit that referenced this pull request Apr 14, 2021

Cherry-pick #24788 to 7.x: Expose processes and their metrics (#25017)

51c860c

Cherry-pick #24788 to 7.x: Expose processes and their metrics (#25017)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ingest Manager] Expose processes and their metrics #24788

[Ingest Manager] Expose processes and their metrics #24788

michalpristas commented Mar 26, 2021

elasticmachine commented Mar 26, 2021 •

edited by jenkins-beats-ci bot

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

simitt left a comment

simitt Mar 29, 2021

simitt Mar 29, 2021

michalpristas commented Mar 29, 2021

ruflin commented Mar 30, 2021

michalpristas commented Mar 31, 2021

michalpristas commented Mar 31, 2021

simitt commented Mar 31, 2021

ruflin commented Mar 31, 2021

ruflin Mar 31, 2021

simitt Mar 31, 2021

michalpristas Mar 31, 2021

ruflin Mar 31, 2021

elasticmachine commented Mar 31, 2021

ruflin Apr 6, 2021

ruflin Apr 6, 2021

michalpristas Apr 6, 2021

ruflin Apr 6, 2021

urso Apr 6, 2021

ruflin Apr 7, 2021

ruflin commented Apr 6, 2021

michalpristas commented Apr 6, 2021 •

edited

urso Apr 6, 2021

urso Apr 6, 2021

urso Apr 6, 2021

urso Apr 7, 2021

michalpristas Apr 8, 2021

[Ingest Manager] Expose processes and their metrics #24788

[Ingest Manager] Expose processes and their metrics #24788

Conversation

michalpristas commented Mar 26, 2021

What does this PR do?

Why is it important?

Checklist

elasticmachine commented Mar 26, 2021 • edited by jenkins-beats-ci bot

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

simitt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michalpristas commented Mar 29, 2021

ruflin commented Mar 30, 2021

michalpristas commented Mar 31, 2021

michalpristas commented Mar 31, 2021

simitt commented Mar 31, 2021

ruflin commented Mar 31, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Mar 31, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Apr 6, 2021

michalpristas commented Apr 6, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Mar 26, 2021 •

edited by jenkins-beats-ci bot

michalpristas commented Apr 6, 2021 •

edited