[Flaky Test]: TestComponentWorkDir – component working directory shouldn't have been modified

[pkoutsovasilis]

Failing test case

TestComponentWorkDir

Error message

component working directory shouldn't have been modified

Build

https://buildkite.com/elastic/elastic-agent/builds/29728#019a395b-81b8-4f86-9d0c-2682d5a6ae37/4036-4351

OS

Windows

Stacktrace and notes

=== FAIL: testing/integration/ess TestComponentWorkDir (62.86s)
    fetcher.go:102: Using existing artifact elastic-agent-9.3.0-SNAPSHOT-windows-x86_64.zip
    fixture.go:311: Extracting artifact elastic-agent-9.3.0-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestComponentWorkDir2441176226
    fixture.go:329: Completed extraction of artifact elastic-agent-9.3.0-SNAPSHOT-windows-x86_64.zip to C:\Users\BUILDK~1\AppData\Local\Temp\TestComponentWorkDir2441176226
    fixture.go:1018: Components were not modified from the fetched artifact
    fixture_install.go:200: [test TestComponentWorkDir] Inside fixture install function
    fixture_install.go:228: [test TestComponentWorkDir] Inside fixture installNoPkgManager function
    fixture_install.go:247: >> running binary with: [C:\Users\BUILDK~1\AppData\Local\Temp\TestComponentWorkDir2441176226\elastic-agent-9.3.0-SNAPSHOT-windows-x86_64\elastic-agent.exe install --force]
    fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
    fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
    fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
    fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
    fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
    fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
    fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
    fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
    fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
    fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
    fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
    fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
    beat_receivers_test.go:1017:
        	Error Trace:	C:/buildkite-agent/builds/bk-agent-prod-gcp-1761900241261597002/elastic/elastic-agent/testing/integration/ess/beat_receivers_test.go:1017
        	Error:      	Not equal:
        	            	expected: time.Date(2025, time.October, 31, 9, 12, 54, 705736000, time.Local)
        	            	actual  : time.Date(2025, time.October, 31, 9, 12, 55, 416799000, time.Local)
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,2 +1,2 @@
        	            	-(time.Time) 2025-10-31 09:12:54.705736 +0000 GMT
        	            	+(time.Time) 2025-10-31 09:12:55.416799 +0000 GMT
        	Test:       	TestComponentWorkDir
        	Messages:   	component working directory shouldn't have been modified
    fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
    fixture.go:812: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe status --output json]
    fixture_install.go:311: [test TestComponentWorkDir] Inside fixture cleanup function
    fixture_install.go:327: collecting diagnostics; test failed
    fixture_install.go:789: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe diagnostics -f C:\buildkite-agent\builds\bk-agent-prod-gcp-1761900241261597002\elastic\elastic-agent\build\diagnostics\TestComponentWorkDir-2025-10-31T09-13-45Z-diagnostics.zip]
    fixture_install.go:740: >> running binary with: [C:\Program Files\Elastic\Agent\elastic-agent.exe uninstall --force]
    fixture.go:1143: Dumping running processes in C:\buildkite-agent\builds\bk-agent-prod-gcp-1761900241261597002\elastic\elastic-agent\build\diagnostics\TestComponentWorkDir-2025-10-31T09-13-45Z-ProcessDump-cleanup.json
    fixture.go:1434: Temporary directory "C:\\Users\\BUILDK~1\\AppData\\Local\\Temp\\TestComponentWorkDir2441176226" preserved for investigation/debugging

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[swiatekm]

The expected time lines up with the log in that test:

{"log.level":"info","@timestamp":"2025-10-31T09:12:54.706Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1787},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

And the actual time lines up with metricbeat starting:

{"log.level":"info","@timestamp":"2025-10-31T09:12:55.416Z","message":"Beat ID: b1d78703-6366-4cda-a2f6-937a27abceee","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"service.name":"metricbeat","ecs.version":"1.6.0","log.origin":{"file.line":843,"file.name":"instance/beat.go","function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure"},"ecs.version":"1.6.0"}

Doesn't look like it's an actual bug, maybe a bad assumption in the test. But I'm not sure why this would only happen on Windows, and only intermittently.

[swiatekm]

Testing a potential fix in #10961

[swiatekm]

Logs indicate the beats process is failing to stop for the following reason:

{"log.level":"error","@timestamp":"2025-10-31T20:32:53.232Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":997},"message":"Spawned new component system/metrics-default: Failed: The handle is invalid.","log":{"source":"elastic-agent"},"component":{"id":"system/metrics-default","state":"FAILED"},"ecs.version":"1.6.0"}

Not sure what can cause the process handle to become invalid this way.

[cmacknz]

The "Handle is invalid" error has shown up in other tests, see #9260 (comment)

[swiatekm]

I'm going to turn on debug logging, improve error reporting in the command runtime, and hopefully we can get a better idea of what exactly the problem is.