Add diagnostics collect command to elastic-agent. #28461

michel-laterman · 2021-10-15T02:44:31Z

What does this PR do?

Add diagnostics collect command to elastic-agent. This option will
create an archive with information that can help with debugging the
elastic-agent. The information includes process metadata (same as the
output of elastic-agent diagnostics), config information, the rendered
policy, and any local log files.

Why is it important?

This command can be used by support to help when debugging issues with the elastic-agent.

Note that it currently DOES NOT redact any credentials, so if it used by a customer, they will need to manually edit the files in the archive to redact credentials.

Contents (config, policies, and metadata) may be written in json or yaml (specified by a flag, default yaml). The archive will have the following structure:

meta/
    elastic-agent-version.yaml
    <process-name>-<routing_key>.yaml
config/
    elastic-agent-local.yaml
    elastic-agent-policy.yaml
logs/
    <all local log files>

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation PR here
[] ~~I have made corresponding change to the default configuration files~~
[] ~~I have added tests that prove my fix is effective or that my feature works~~
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Run the agent normally, then run elastic-agent diagnostics collect.

Related issues

Relates Beats diagnostics tooling #21965

Add diagnostics collect command to elastic-agent. This option will create an archieve with information that can help with debugging the elastic-agent. The information includes process metadata (same as the output of elastic-agent diagnostics), config information, the rendered policy, and any local log files.

elasticmachine · 2021-10-15T02:44:33Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

michel-laterman · 2021-10-15T02:49:15Z

x-pack/elastic-agent/pkg/agent/cmd/diagnostics.go

+		return err
+	}
+
+	// using Data() + "/logs", for some reason default paths/Logs() is the home dir...


I'm confused as to why path.Logs() returned <path> instead of <path>/data/logs (as I expected), anyone have insight into this?

logPath defaults to topPath which is what you see if no override is provided x-pack/elastic-agent/pkg/agent/application/paths/common.go:L40

michel-laterman · 2021-10-15T02:50:59Z

x-pack/elastic-agent/pkg/agent/cmd/diagnostics.go

+		if fErr != nil {
+			return fmt.Errorf("unable to walk log dir: %w", fErr)
+		}
+		name := strings.TrimPrefix(path, logPath)


name in this case gives the name of the directory/file within log/ to write into the archive. For exampledefault/filebeat-json.log

michel-laterman · 2021-10-15T02:53:17Z

x-pack/elastic-agent/pkg/agent/cmd/diagnostics.go

+	}
+	zw := zip.NewWriter(f)
+
+	zf, err := zw.Create("meta/")


the docs for zip indicate that ending a name with a slash creates a directory.

elasticmachine · 2021-10-15T02:58:05Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 98 min 49 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.

michalpristas · 2021-10-15T09:54:05Z

x-pack/elastic-agent/CHANGELOG.next.asciidoc

@@ -147,4 +147,5 @@
 - Support ephemeral containers in Kubernetes dynamic provider. {issue}27020[#27020] {pull}27707[27707]
 - Add complete k8s metadata through composable provider. {pull}27691[27691]
 - Add diagnostics command to gather beat metadata. {pull}28265[28265]
+- Add diagnostics collect command to gather beat metadata, config, policy, and logs and bundle it into an archieve. {pull}28461[28461]


s/archieve/archive

michalpristas

looks good, please take a look at these small issues i raised and test on windows to check processing of path separator

michalpristas · 2021-10-15T10:03:41Z

x-pack/elastic-agent/pkg/agent/cmd/diagnostics.go

+		return closeHandlers(err, zw, f)
+	}
+
+	zf, err = zw.Create("meta/elastic-agent-version." + outputFormat)


tested on windows as well? i'm wondering because of / vs \ as a path separator.
same thing all over this file

michalpristas · 2021-10-15T10:06:09Z

x-pack/elastic-agent/pkg/agent/cmd/diagnostics.go

+	// using Data() + "/logs", for some reason default paths/Logs() is the home dir...
+	logPath := filepath.Join(paths.Data(), "logs") + string(filepath.Separator)
+	return filepath.WalkDir(logPath, func(path string, d fs.DirEntry, fErr error) error {
+		if fErr != nil {


let's hande IsNotExist(err) and continue without processing if so. i've had issues with Walk before and this is not serious but can break the process

michalpristas · 2021-10-15T10:07:20Z

x-pack/elastic-agent/pkg/agent/cmd/diagnostics.go

+			return nil
+		}
+
+		lf, err := os.Open(path)


not closed, better to extract to a func and use defer to close the file

done, just added closeHandlers to my exit points

michalpristas · 2021-10-15T10:09:04Z

x-pack/elastic-agent/pkg/agent/cmd/diagnostics.go

+
+// writeFile writes json or yaml data from the interface to the writer.
+func writeFile(w io.Writer, outputFormat string, v interface{}) error {
+	if outputFormat == "yaml" {


not serious at all but we default to yaml when using a command, we could also use json here if specified and default to yaml as well

fix typos, handle file not exist errors, close open file handles for logs after copying, writeFile will default to yaml instead of json behaviour.

michel-laterman · 2021-10-16T02:45:24Z

I've tested on windows now and found and fixed an issue with the archive file name (Windows does not allow : in file names).

Currently the command runs and creates an archive, however it's not able to collect logs (on Windows).
The log path I create from paths.Data() + "logs" (default c:\program files\elastic\agent\data\logs) does not exist on windows. The logs are in a sub-directory of data: c:\program files\elastic\agent\data\elastic-agent-<hash>\logs.
I can't find a way to find this path with the local config settings.logging.files.path or path.Logs().
@michalpristas, @blakerouse any suggestions?

michalpristas · 2021-10-17T05:46:36Z

Home should point to versioned home directory

michel-laterman · 2021-10-18T20:24:49Z

/test

michel-laterman · 2021-10-19T00:04:27Z

/package

Add diagnostics collect command to elastic-agent. Add diagnostics collect command to elastic-agent. This option will create an archieve with information that can help with debugging the elastic-agent. The information includes process metadata (same as the output of elastic-agent diagnostics), config information, the rendered policy, and any local log files. (cherry picked from commit 99ebf3e)

Add diagnostics collect command to elastic-agent. Add diagnostics collect command to elastic-agent. This option will create an archieve with information that can help with debugging the elastic-agent. The information includes process metadata (same as the output of elastic-agent diagnostics), config information, the rendered policy, and any local log files. (cherry picked from commit 99ebf3e) Co-authored-by: Michel Laterman <82832767+michel-laterman@users.noreply.github.com>

Add diagnostics collect command to elastic-agent. Add diagnostics collect command to elastic-agent. This option will create an archieve with information that can help with debugging the elastic-agent. The information includes process metadata (same as the output of elastic-agent diagnostics), config information, the rendered policy, and any local log files.

Add diagnostics collect command to elastic-agent. Add diagnostics collect command to elastic-agent. This option will create an archieve with information that can help with debugging the elastic-agent. The information includes process metadata (same as the output of elastic-agent diagnostics), config information, the rendered policy, and any local log files. (cherry picked from commit 99ebf3e)

michel-laterman added enhancement backport-v7.16.0 Automated backport with mergify Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team labels Oct 15, 2021

michel-laterman requested review from michalpristas and blakerouse October 15, 2021 02:44

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 15, 2021

michel-laterman commented Oct 15, 2021

View reviewed changes

Fix missing return, add comments, fix spelling

4dcbc4f

michalpristas reviewed Oct 15, 2021

View reviewed changes

michalpristas approved these changes Oct 15, 2021

View reviewed changes

michel-laterman added 2 commits October 15, 2021 11:24

review feedback, close log file handles

043c8c8

fix typos, handle file not exist errors, close open file handles for logs after copying, writeFile will default to yaml instead of json behaviour.

Fix archive file name for windows

4766260

Change paths.Data() to paths.Home() to work across platforms

500105a

Merge remote-tracking branch 'origin/master' into diagnostics-collect

f434c20

michel-laterman merged commit 99ebf3e into elastic:master Oct 19, 2021

michel-laterman deleted the diagnostics-collect branch October 19, 2021 15:31

mergify bot mentioned this pull request Oct 19, 2021

[7.x](backport #28461) Add diagnostics collect command to elastic-agent. #28535

Merged

mdelapenya mentioned this pull request Oct 20, 2021

[Agent] Elastic-Agent-Packaging-Linux is failed (there for e2e-testing pr support) which blocks merges - seems to be a Heartbeat container build problem #28570

Closed

michel-laterman added the v8.0.0 label Nov 10, 2021

mergify bot mentioned this pull request Nov 10, 2021

[7.16](backport #28461) Add diagnostics collect command to elastic-agent. #28920

Closed

P1llus mentioned this pull request Nov 15, 2021

Libbeat diagnostic tooling #28977

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add diagnostics collect command to elastic-agent. #28461

Add diagnostics collect command to elastic-agent. #28461

michel-laterman commented Oct 15, 2021 •

edited

elasticmachine commented Oct 15, 2021

michel-laterman Oct 15, 2021

michalpristas Oct 15, 2021

michel-laterman Oct 15, 2021

michel-laterman Oct 15, 2021

elasticmachine commented Oct 15, 2021 •

edited by jenkins-beats-ci bot

Build stats

michalpristas Oct 15, 2021

michalpristas left a comment

michalpristas Oct 15, 2021

michalpristas Oct 15, 2021

michalpristas Oct 15, 2021

michel-laterman Oct 15, 2021

michalpristas Oct 15, 2021

michel-laterman commented Oct 16, 2021

michalpristas commented Oct 17, 2021

michel-laterman commented Oct 18, 2021

michel-laterman commented Oct 19, 2021

Add diagnostics collect command to elastic-agent. #28461

Add diagnostics collect command to elastic-agent. #28461

Conversation

michel-laterman commented Oct 15, 2021 • edited

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

elasticmachine commented Oct 15, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Oct 15, 2021 • edited by jenkins-beats-ci bot

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

Choose a reason for hiding this comment

michalpristas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michel-laterman commented Oct 16, 2021

michalpristas commented Oct 17, 2021

michel-laterman commented Oct 18, 2021

michel-laterman commented Oct 19, 2021

michel-laterman commented Oct 15, 2021 •

edited

elasticmachine commented Oct 15, 2021 •

edited by jenkins-beats-ci bot