Errors in CI build running docker command (fork/exec errors) #2537
Description
There are some steps in CI builds in the integrations repository that fail related to an error trying to execute a docker command.
Mainly related to unable to determine Docker compose version in the context of fetching the logs after running the tests.
Examples of these errors:
Error: error running package system tests: could not complete test run: dump failed: can't dump Elastic stack logs: can't fetch service logs (service: elasticsearch): could not create docker compose project: unable to determine Docker Compose version: running Docker Compose version command failed: failed to start command with pseudo-tty: fork/exec /usr/bin/docker: operation not permitted
Error: error running package system tests: could not complete test run: dump failed: can't dump Elastic stack logs: can't copy internal logs (service: fleet-server): could not create docker compose project: unable to determine Docker Compose version: running Docker Compose version command failed: failed to start command with pseudo-tty: fork/exec /usr/bin/docker: operation not permitted
Some Buildkite builds where this error is raised in the last days:
- https://buildkite.com/elastic/integrations/builds/24642#01961ebf-80c7-4932-8a90-7e6ed688afb6/696-2654
- https://buildkite.com/elastic/integrations/builds/24658#01962000-5d7b-4061-b499-c122369ad6ea/669-1180
- https://buildkite.com/elastic/integrations/builds/24629#01961d5d-3a56-4c22-a19e-c935986ecaef/642-1051
- https://buildkite.com/elastic/integrations/builds/24630#01961d5c-c216-4faa-b69f-11787254883c/601-1227
- https://buildkite.com/elastic/integrations/builds/24624#01961c86-a365-4cfe-8e84-7e03897a9c9d/643-1629
- https://buildkite.com/elastic/integrations/builds/24614#01961aa7-1908-4498-93a4-ae3c53709fe9/636-1547
- https://buildkite.com/elastic/integrations/builds/24622#01961c68-2c27-4269-ad06-1487f2bba5df/647-2581
- https://buildkite.com/elastic/integrations/builds/24622#01961c68-2c44-4393-8f2b-54f553aea821/624-1463
- https://buildkite.com/elastic/integrations/builds/24590#019619a7-8a8b-475f-9111-ab78e43013ce/621-1291
- https://buildkite.com/elastic/integrations/builds/24590#019619a7-8aa8-4d83-bd3c-9fe42f6ec27e/591-848
This error is not raised in elastic-package CI builds.
This is related to the usage of PTY in docker compose commands. Related PRs:
- Return stderr as part of the error when compose fails #1801
- Return stderr as part of the error when compose fails #1801
- Refactor PTTY handling in compose commands #1924
For now this dependency is set to be ignored for this current minor version:
Related also to: creack/pty#196
Tried to repeat the execution of the failed command in case of that kind of errors in #2534, but the error keeps being raised. The difference is that in the second iteration the message changes from operation not permitted to bad file descriptor (Buildkite build):
2025/04/11 10:32:46 DEBUG Dump stack logs for elastic-agent
2025/04/11 10:32:47 DEBUG running command: /usr/bin/docker compose version --short
2025/04/11 10:32:47 DEBUG Determined Docker Compose version: 2.24.1
2025/04/11 10:32:47 DEBUG running command: /usr/bin/docker compose -f /opt/buildkite-agent/.elastic-package/profiles/default/stack/docker-compose.yml --progress plain -p elastic-package-stack logs elastic-agent
2025/04/11 10:32:47 DEBUG Repeating docker command (failure fork/exec): /usr/bin/docker compose version --short (Error: fork/exec /usr/bin/docker: operation not permitted)
2025/04/11 10:32:48 DEBUG Repeating docker command (failure fork/exec): /usr/bin/docker compose version --short (Error: fork/exec /usr/bin/docker: bad file descriptor)
2025/04/11 10:32:49 DEBUG Repeating docker command (failure fork/exec): /usr/bin/docker compose version --short (Error: fork/exec /usr/bin/docker: bad file descriptor)
2025/04/11 10:32:50 DEBUG Repeating docker command (failure fork/exec): /usr/bin/docker compose version --short (Error: fork/exec /usr/bin/docker: bad file descriptor)
2025/04/11 10:32:51 DEBUG Repeating docker command (failure fork/exec): /usr/bin/docker compose version --short (Error: fork/exec /usr/bin/docker: bad file descriptor)
2025/04/11 10:32:52 DEBUG Uninstalling package...
2025/04/11 10:32:52 DEBUG GET https://127.0.0.1:5601/api/fleet/epm/packages/windows
2025/04/11 10:32:52 DEBUG DELETE https://127.0.0.1:5601/api/fleet/epm/packages/windows/2.5.2
Error: error running package system tests: could not complete test run: dump failed: can't dump Elastic stack logs: can't copy internal logs (service: elastic-agent): could not create docker compose project: unable to determine Docker Compose version: running Docker Compose version command failed: failed to start command with pseudo-tty: fork/exec /usr/bin/docker: bad file descriptor