Skip to content

fix(webserver): tolerate a stale non-FIFO /var/tmp/logpipe on container start (#8635) [skip ci] - #8635

Merged
rfay merged 2 commits into
ddev:mainfrom
rfay:20260730_rfay_fix_logpipe_race
Jul 31, 2026
Merged

fix(webserver): tolerate a stale non-FIFO /var/tmp/logpipe on container start (#8635) [skip ci]#8635
rfay merged 2 commits into
ddev:mainfrom
rfay:20260730_rfay_fix_logpipe_race

Conversation

@rfay

@rfay rfay commented Jul 30, 2026

Copy link
Copy Markdown
Member

The Issue

The all-project-types job failed in TestDbServer with:

+ set -eu -o pipefail
+ logpipe=/var/tmp/logpipe
+ [[ ! -p /var/tmp/logpipe ]]
+ mkfifo /var/tmp/logpipe
mkfifo: cannot create fifo '/var/tmp/logpipe': File exists

which made the ddev-webserver container exit immediately, so ddev start
timed out waiting for it to become healthy. A restart of the job made it
pass.

How This PR Solves The Issue

The ddev-webserver container's actual startup command is /pre-start.sh
(set via command: /pre-start.sh in app_compose_template.yaml). Its only
job is to create a FIFO at /var/tmp/logpipe and cat it forever as PID 1;
DDEV's Go code later docker execs /start.sh into the running container,
redirecting its output into that same pipe so it shows up in docker logs.
Both start.sh variants use the identical pattern for the same reason
(so /var/tmp/logpipe also works for non-DDEV/standalone container usage).

All three copies used a non-atomic check-then-create:

if [[ ! -p ${logpipe} ]]; then
    mkfifo ${logpipe}
fi

In the failing run, the test had just stopped/removed the
ddev-TestPkgWordpress-web container (finishing a mariadb 11.8 iteration)
and immediately recreated a brand-new container of the same name a couple
of seconds later for a mariadb 12.3 iteration. /var/tmp is only the
container's own writable layer (not a bind mount or named volume), so a
genuinely fresh container should never have anything at that path already.
The fact that something was there — and wasn't recognized as a valid FIFO —
points to a Docker/overlay2 storage-layer race on the GitHub-hosted runner,
where the previous container's writable layer wasn't fully torn down before
the new one (same name) was created, so it inherited stale leftover state.

This doesn't fix that underlying container-runtime race (which is outside
DDEV's control), but it makes the three scripts tolerant of it by removing
any stale non-pipe file before calling mkfifo:

if [[ ! -p ${logpipe} ]]; then
    rm -f "${logpipe}"
    mkfifo "${logpipe}"
fi

WebTag in pkg/versionconstants/versionconstants.go is bumped to match,
since the ddev-webserver container scripts changed.

Manual Testing Instructions

  1. make linux_amd64 (or your platform) and confirm the build succeeds.
  2. ddev start a project and confirm the web container comes up healthy as
    usual -- this is a defensive fix for a rare race, so normal startup is
    unaffected.
  3. To directly exercise the fix, exec into a running web container and
    simulate the stale-file condition before a restart:
    docker exec ddev-<project>-web sh -c 'rm -f /var/tmp/logpipe && touch /var/tmp/logpipe'
    ddev restart
    
    Before this change, the container would fail to become healthy
    (mkfifo: File exists). After this change, it starts normally.

Automated Testing Overview

No new automated test is added -- this addresses a rare Docker
storage-layer race on CI runners that isn't reliably reproducible in a
unit/integration test. Existing ddevapp integration tests (e.g.
TestDbServer) exercise the affected startup path on every run.

Release/Deployment Notes

Bumps the ddev-webserver image tag (WebTag). No user-facing behavior
change under normal conditions; only affects the rare case where a stale
/var/tmp/logpipe file exists at container start.

@github-actions github-actions Bot added bugfix dependencies Pull requests that update a dependency file labels Jul 30, 2026
@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown

@rfay

rfay commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

History of the /var/tmp/logpipe mechanism

While digging into this bug I traced the pre-start.sh / start.sh FIFO-logging pattern back to its origin, since its original stated justification (Gitpod) no longer applies to DDEV. Recording the history here for anyone who finds this later.

Date Commit PR Summary
2023-05-13 d957ea71f #4895 (fixes #4889) Introduced the mechanism. Replaced command: sleep infinity with pre-start.sh as PID 1, adding the /var/tmp/logpipe FIFO + SIGTERM trap, because writing directly to /proc/1/fd/1 from a docker exec'd process didn't work on Gitpod (gitpod-io/gitpod#17551).
2023-07-10 f39c884c6 #5107 Fixed a typo in the trap (kill -- --1 -> kill -- -1) so the "kill everything" handler actually worked.
2024-11-11 bc338507e #6706 Unrelated portability fix: #!/bin/bash -> #!/usr/bin/env bash.
2025-10-21 889558ea7 #7716 Removed all Gitpod support from DDEV. Left pre-start.sh's Gitpod-referencing comment in place as acknowledged historical color; did not touch the FIFO mechanism itself.
2026-05-08 3d29d0ae0 #8396 (fixes #8295) Fixed a different bug in the same mechanism: cat running in the foreground deferred bash's SIGTERM trap handling, so ddev stop took the full 10s grace period before Docker SIGKILLed the container. Backgrounding cat (cat < pipe & ; wait) dropped stop time to ~1.4s.
2026-07-30 2a2c1ce0b #8635 (this PR) Fixed the mkfifo race this PR addresses: a non-atomic check-then-create ([[ ! -p ]] then mkfifo) failed with "File exists" when a stale non-FIFO file was left at /var/tmp/logpipe, apparently from a Docker/overlay2 storage-layer race on the CI runner recreating a same-named container back-to-back.

Is the mechanism still appropriate?

Given Gitpod is gone, it's fair to ask whether this whole apparatus is now legacy complexity that should be reverted to the pre-2023 sleep infinity design. It isn't purely that: /var/tmp/logpipe is also the stdout_logfile target for every supervisord-managed process (apache.conf, php-fpm.conf, supervisord-nginx-fpm.conf), so it's the general log fan-in mechanism for the container, not just a wrapper around the one docker exec /start.sh call. #8396 landed after Gitpod removal and shows real, current value (10s -> 1.4s stop time) still being extracted from it. So the design itself still looks right; it's just been a fragile ~25 lines that has needed three narrow fixes over its life, of which this PR is the third.

@rfay
rfay marked this pull request as ready for review July 30, 2026 21:10
@rfay
rfay requested a review from a team as a code owner July 30, 2026 21:10
@rfay
rfay requested a review from stasadev July 30, 2026 21:10
…er start

The web container's logpipe setup (`pre-start.sh` and both `start.sh`
variants) checked `[[ ! -p ${logpipe} ]]` before calling `mkfifo`, but did
not remove a pre-existing non-pipe file at that path first. On a freshly
created container this should never happen, since `/var/tmp` is only the
container's own writable layer, but a Docker storage-layer race on CI
runners (containers stopped/recreated back-to-back with the same name)
occasionally left stale content there, causing `mkfifo` to fail with
"File exists" and the container to exit immediately.

`rm -f` the path before `mkfifo` so a stale non-pipe file no longer
crashes container startup.

Bumps WebTag since the ddev-webserver container scripts changed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@rfay
rfay force-pushed the 20260730_rfay_fix_logpipe_race branch from 2a2c1ce to a653ee1 Compare July 30, 2026 23:12
@rfay
rfay force-pushed the 20260730_rfay_fix_logpipe_race branch from a653ee1 to 67ec69b Compare July 30, 2026 23:20
@rfay

rfay commented Jul 30, 2026

Copy link
Copy Markdown
Member Author

New image pushed, rebased after mysql 9.7 PR

@stasadev stasadev left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@rfay rfay changed the title fix(webserver): tolerate a stale non-FIFO /var/tmp/logpipe on container start fix(webserver): tolerate a stale non-FIFO /var/tmp/logpipe on container start (#8635) [skip ci] Jul 31, 2026
@rfay
rfay merged commit efda041 into ddev:main Jul 31, 2026
49 checks passed
@rfay
rfay deleted the 20260730_rfay_fix_logpipe_race branch July 31, 2026 14:35
rfay added a commit to weitzman/ddev that referenced this pull request Jul 31, 2026
…er start (ddev#8635) [skip ci]

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
rfay added a commit to rfay/ddev that referenced this pull request Aug 4, 2026
Any change under containers/ddev-webserver/ needs a matching WebTag bump
in versionconstants.go so ddev actually pulls an image built from this
branch instead of the one WebTag previously pointed at, matching the
convention used by prior webserver-only fixes (e.g. ddev#8635).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix dependencies Pull requests that update a dependency file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants