GitHub Community
Deployment fails in a custom code-only guard, followed by “Matching delimiter not found 'EOF'” #212388
Replies: 7 comments 29 replies
|
Short answer to (1): yes, the Why it happens
echo 'stdout<<EOF' >> "$GITHUB_OUTPUT"
drone-ssh ... | tee -a "$GITHUB_OUTPUT"
echo 'EOF' >> "$GITHUB_OUTPUT"If the remote script exits non-zero, So the real failure is the one before it: the Preserving the original error and exit status
set +e
ssh deploy@"$HOST" 'bash -s' < scripts/deploy.sh 2>&1 | tee deploy.log
rc=${PIPESTATUS[0]}
delim="EOF_$(openssl rand -hex 8)" # random delimiter so log content can't end the block early
{ echo "rc=$rc"; echo "log<<$delim"; cat deploy.log; echo "$delim"; } >> "$GITHUB_OUTPUT"
exit "$rc"
(2) Minimal credential-free reproduction Step 1 reproduces the exact message without SSH at all: name: eof-repro
on: workflow_dispatch
jobs:
repro:
runs-on: ubuntu-latest
steps:
- shell: bash
run: |
set -euo pipefail
echo 'stdout<<EOF' >> "$GITHUB_OUTPUT"
(echo "guard failed"; exit 3) | tee -a "$GITHUB_OUTPUT"
echo 'EOF' >> "$GITHUB_OUTPUT"Step 2 isolates the action: run Step 3 isolates your script: run the guard directly on a copy of the host layout (container or VM) with (3) Recovering from a partial Docker Compose deployment
Happy to look at a sanitized excerpt of the workflow and guard script if you post one. |
|
Thank you. We now have a concrete update:
No compatibility rules were relaxed, and nothing was merged or deployed. Production Deploy #253 remains unexplained. Given this earlier failure stage, would you prioritize isolating the fingerprint-helper failure before expanding the schema-helper matrix? |
|
Yes, I'd isolate the fingerprint helper first. The guard stops at the first failing stage, so in that pipeline the schema helper never runs until One thing to keep separate, though: which candidate did that CI run use? If it was the newer draft, the failure may simply be the git diff --name-only <baseline-rev> <deploy-253-release-rev>If Isolating the helper failure
For #253 specifically The fingerprint helper runs with |
|
Thanks. We now have the completed CI result for PR #176 at commit 60fcc7e326826677a4c5d638bfec3c55d18b9a70: CODE_ONLY_GUARD_FAIL stage=live-source code=SOURCE_BLOB_UNREVIEWED The exact rejection-code relay worked. All 41 backup/orchestration/diagnostic tests, synthetic PostgreSQL restore checks, equipment-count checks, and ten independent acceptance jobs passed. The overall CI still failed at the source guard, before the schema helper. The fingerprint helper checks /app inside the image; stdin supplies the helper code, not the source tree. A source/image matrix would therefore need explicit source fixtures. Given this confirmed rejection, what is the smallest useful next isolated test? Would you first compare #253’s exact baseline/release protected files, or test the real helper invocation and stdin/output contract? Deploy #253’s original cause remains unknown. Neither draft PR has been merged or deployed, and we have not changed the compatibility rules or production marker. CI: https://github.com/wudenny6126/-etao-management-platform/actions/runs/42451313422 |
|
Thanks for your help. We implemented deployment phase journals and fixed a disconnect-handling race in draft PR #176, commit 27284c3. The corresponding backup/journal checks, synthetic PostgreSQL restore and equipment-count checks passed. At our last check, overall CI was still running. Nothing has been merged or deployed. The journal records stage start/completion and retains incomplete-operation protection on failure. Baseline/image metadata remains unknown, and host completion is explicitly separate from later public HTTPS verification. Before adding more recovery functionality, what essential evidence or failure case is still missing from this design? We are seeking source review and synthetic testing guidance only. Production diagnosis, marker clearing and weakening compatibility checks remain outside scope. Deploy #253’s original cause remains unknown. |
|
Catching up on your last two updates here, plus the container-ID question in Mohit's thread. Smallest next isolated test: compare #253's protected files first It needs nothing but git, and its result tells you which test is worth building next. # for each path the allowlist pins
git diff --name-only <baseline-rev> <deploy-253-release-rev> -- <protected paths>
git rev-parse <deploy-253-release-rev>:<path> # compare with the allowlisted blob IDs
One caveat, since the helper reads If you do need the invocation test, a minimal stdin/output contract test is three tiny fixture images (
The third row is what keeps a crashed helper from being reported as a policy rejection. Essential evidence still missing from the journal design Mohit's two tests cover the failure cases. The evidence gap is the one your journal already lists: "baseline/image metadata remains unknown." Without it, recovery can't prove it returned to the baseline. Before the overlay starts, record:
Also write the journal entry before each step, not after. A crash mid-step should then leave "started, not completed", which recovery treats as unknown state, never as done. On the container-ID question (here) Not Mohit, but yes: the container ID is expected to change, and recording it for traceability is the right call. I'd define success as matching:
|
|
Thanks, that helps. From what you describe, production is in a split state: the containers and database are still on the earlier revision, but the source checkout and Step 1: Freeze and snapshot (read-only) Until this is resolved, don't run Record the current state: cat /path/to/unfinished.json
git -C /path/to/app rev-parse HEAD
git -C /path/to/app status --short
docker compose ps
docker inspect --format '{{.Name}} running={{.Image}} tag={{.Config.Image}}' $(docker compose ps -q)
docker image inspect --format '{{.Id}}' <image tag used in compose>Compare the last two outputs. If the candidate build reused the tag the service runs, that tag now points to the candidate image, and any container recreate would start it. Then take a database backup now, since the deploy never reached its own backup step (e.g. Step 2: Identify the failed check Run the guard outside the workflow, in an isolated copy, with the same inputs as the failed run: the running revision, the candidate revision and the env file. Read the guard first to confirm it is read-only (no writes, no marker changes, no docker commands) before running it anywhere near production. Make it name the cause while still failing closed. If it's Python with one broad import sys, traceback
for name, check in CHECKS: # e.g. [("compat_fingerprint", check_fingerprint), ...]
try:
check()
except Exception:
print(f"GUARD_FAIL check={name}", file=sys.stderr)
traceback.print_exc()
sys.exit(1)For a shell guard, Given your earlier note that the candidate doesn't match the compatibility fingerprint in CI, that's the first check I'd suspect, but confirm it with the instrumented run rather than assume it. Step 3: Bring the host files back in line with what's running The safer direction is to make the files match the containers, not the other way round:
Step 4: Resolve the marker with a record Only after Step 3 checks out. Don't delete the marker; archive it (e.g. Step 5: Retry through the normal path
If you post the guard's list of checks and its |
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as spam.
|
Thanks for providing the context. I would investigate this as two separate issues rather than treating both messages as one failure. 1. Understanding the EOF errorYes, the EOF error can be a secondary GitHub Actions output-capture failure. When If the remote command exits with a non-zero status,
This behavior is documented in [appleboy/ssh-action issue #416](appleboy/ssh-action#416). A temporary workaround is to disable The permanent fix should ensure that the output delimiter is always closed while preserving the original command's exit status. For example: if [[ "$INPUT_CAPTURE_STDOUT" == "true" ]]; then
echo 'stdout<<EOF' >> "$GITHUB_OUTPUT"
set +e
"$TARGET" "$@" | tee -a "$GITHUB_OUTPUT"
status="${PIPESTATUS[0]}"
set -e
echo 'EOF' >> "$GITHUB_OUTPUT"
exit "$status"
else
"$TARGET" "$@"
fiThis allows the output to be finalized before returning the original failure code. 2. Minimal reproductionThis can be reproduced without an SSH connection or production credentials: output=$(mktemp)
GITHUB_OUTPUT="$output" bash -c '
set -euo pipefail
echo "stdout<<EOF" >> "$GITHUB_OUTPUT"
(echo "Simulated remote failure"; exit 23) |
tee -a "$GITHUB_OUTPUT"
echo "EOF" >> "$GITHUB_OUTPUT"
'
echo "Captured output file:"
cat "$output"The script exits before writing the closing delimiter. This demonstrates the output-capture failure independently of SSH. For an end-to-end test, use a disposable SSH container with a temporary key and a pinned action version. Run a harmless command that prints a message and exits with status 23, first with This will help distinguish an action-level output problem from an actual failure in the remote deployment script. 3. Safe recovery for partial Docker Compose deploymentsI would recommend treating deployment as a controlled state machine:
For recovery, inspect the actual running containers, image digests, service health and database state. Do not assume that the source checkout represents the running application version. Rollback should use a known-good deployment configuration and compatible images. Persistent volumes and database data must remain untouched. Database migrations should also be designed to support safe rollback or forward recovery. Most importantly, the incomplete marker should only be cleared after a recovery or reconciliation procedure confirms the actual system state. It should not be removed simply to allow another deployment. Additional recommendationI would also make the custom code-only guard report the exact failed condition rather than a generic failure message. Log the check identifier and sanitized expected-versus-actual values. The compatibility fingerprint mismatch should remain a separate preflight failure. It should not be bypassed as part of recovering the original deployment. This approach preserves the original failure information, makes the SSH output issue independently testable, and avoids accidentally turning a partially completed deployment into a data-loss incident. |
|
Thank you. Issue #416 helps explain how output capture can produce a secondary EOF error, although we still cannot confirm that mechanism for Deploy #253. Our latest progress:
Both PRs remain drafts, with no merge or deployment. Production state and backup recoverability remain unverified. Your phase-journal and recovery suggestions address our remaining gap. Could you recommend the minimum journal fields and isolated recovery acceptance tests for a deployment that overlays host files before replacing containers? We will retain the interlock and compatibility checks. Any reproduction or recovery rehearsal will remain isolated from production. |
|
Thanks for your help. We implemented deployment phase journals and fixed a disconnect-handling race in draft PR #176, commit 27284c3. The corresponding backup/journal checks, synthetic PostgreSQL restore and equipment-count checks passed. At our last check, overall CI was still running. Nothing has been merged or deployed. The journal records stage start/completion and retains incomplete-operation protection on failure. Baseline/image metadata remains unknown, and host completion is explicitly separate from later public HTTPS verification. Before adding more recovery functionality, what essential evidence or failure case is still missing from this design? We are seeking source review and synthetic testing guidance only. Production diagnosis, marker clearing and weakening compatibility checks remain outside scope. Deploy #253’s original cause remains unknown. |
|
Thanks for the update. I think the phase journal is a good improvement. Before adding more recovery functionality, I would focus on one missing piece: proving that the previous deployment can actually be reconstructed after a partial host-file update. I would recommend two final isolated tests: 1. Partial host-overlay recovery
2. Journal consistency and interrupted recovery
For acceptance evidence, I would retain:
The key acceptance criterion should be that recovery is repeatable, does not lose persistent data, and never clears the interlock while the actual deployment state remains uncertain. I would prioritize these tests over adding further recovery features. If the previous baseline cannot be positively identified or restored, the deployment should remain blocked. This can all be demonstrated in an isolated environment without accessing or modifying production. |
|
Thanks—this identifies a gap in our current tests. Our Compose fixture tests cover container switching and failure handling, but do not yet prove baseline reconstruction or repeatable recovery after interruption. We plan to add both scenarios in isolation, preserving the previous file contents, permissions and configuration, and checking actual container/image identities and synthetic database integrity. One clarification: after recovery recreates a baseline service, its container ID will change. Would you consider recovery successful when the baseline image ID, configuration, mounts and health match, persistent data is unchanged, and the new container ID is recorded? We will keep the interlock while any required evidence is missing. No production action or protection bypass is planned. |
|
Yes, I would consider that a successful recovery. A recreated container is expected to receive a new container ID, so requiring the original container ID would be an incorrect acceptance criterion. I would use the following checks:
One distinction for database services: unchanged data should be expected when no migration or data modification was intended. Otherwise, verify the expected schema version and data integrity rather than requiring identical data. I would therefore treat recovery as successful when the baseline's required behavior, configuration and persistent state are demonstrably restored, regardless of the new container ID. The interlock should only be cleared after all required checks pass and the recovery outcome is durably recorded. |
|
One extra recovery test: exercise the missing-image case. In the isolated fixture, remove the local baseline image and make the registry unavailable before recovery. An image ID or digest in the journal identifies the baseline, but does not make its bytes available. If there is no verified copy, recovery should leave the interlock in place and report which image is missing, rather than falling back to the current tag. Then restore the exact baseline image from a retained copy and rerun recovery. This tests whether the rollback dependency is actually available, not just whether the journal has the right identifiers. |
|
Thanks for suggesting the missing-image test. We ran it in an isolated environment. When the previous image was unavailable, recovery stopped and kept the interlock in place. After we reloaded the retained image and verified its exact image ID, the isolated recovery succeeded. The production deployment stopped at the code-only guard after updating the host files. The log does not show which guard check failed. Separately, our equipment changes passed the concurrent-update and audit tests in CI, but the candidate still failed the release guard and has not been deployed. Could you review our proposed recovery criteria? We plan to require evidence that the intended baseline image is available and matches its recorded ID, that the deployed configuration and persistent data are consistent with that baseline, and that the application passes its health checks. Which of these checks would you require before resuming normal releases, and is there anything important missing? We’re looking for a review of the recovery plan based on existing records and the isolated test results. |
|
Thanks for running the missing-image test. If you can post a sanitized recovery checklist, I’d be happy to review it. The main thing I’d check is how you verify the running containers, host configuration, and persistent data against the last known-good baseline, especially since the failed deployment had already updated the host files. I’d also keep baseline recovery separate from approval to deploy the new candidate. |
|
Thank you for offering to review our recovery checklist. Our immediate goal is to resolve an unfinished deployment marker without interrupting the running application or losing data employees have entered since the failed deployment. The failed run stopped at the code-only guard before its backup and application-container replacement steps, but it may already have overlaid host files and installed configuration. We have not confirmed the current production state or a recovery procedure compatible with the installed supervisor. What we have done so far: • Reviewed the existing Actions logs and deployment-script order. The guard reported only a generic failure, so we cannot identify its first failing check from those logs. Our main difficulties are identifying the exact running baseline, confirming current configuration and data state, and establishing a verified recovery procedure for the installed supervisor. The marker remains unresolved according to the available evidence. Could you help us assess:
We would especially value specific acceptance criteria, stop conditions, and the smallest practical path to recovery. We will keep recovery of the existing service separate from approval of a new release. |
|
Thanks for the detailed write-up. I’d keep the application and database running and start with a read-only inventory. The first decision is whether the current service can be verified against a known-good baseline, not whether the marker can be cleared.
|
|
Based on the symptoms, I would investigate the remote deployment failure separately from the EOF output-capture error. 1. Preserving the original SSH error Yes, the I recommend:
Do not assume the EOF message is the root cause until a minimal reproduction confirms it. 2. Minimal reproduction Create a separate test repository with:
Compare the remote exit status, stdout, stderr, and GitHub Actions logs. This should help isolate an action-level output issue from custom deployment-script behavior. 3. Safe Docker Compose recovery I recommend implementing a deployment state machine rather than treating deployment as a single shell operation. Suggested states:
Any failure should transition to:
Before another deployment is allowed:
Avoid automatically running One additional recommendation: Keep source checkout, deployed image digest, and running container version as separate records. A successful Git checkout does not prove that the corresponding application version is running. I would also investigate issue #375 against the exact action version in use before changing the deployment logic. The safest approach is to reproduce the failure in an isolated environment, identify the original failing command, and test recovery independently before considering any production deployment. |
|
Thank you for your suggestions. We checked the exact action used in Deploy #253: appleboy/ssh-action commit 0ff4204d59e8e51228ff73bce53f80d53301dee2 with drone-ssh 1.8.2. Its entrypoint opens a multiline output block before running SSH under set -euo pipefail. A nonzero SSH exit appears able to stop the script before the closing EOF is written. Our log shows the code-only guard failure before the EOF error. This seems consistent with EOF being secondary, but it does not identify which guard check failed. What we have done so far: • Reviewed the existing Actions log and the exact action entrypoint used by that run. Our difficulty is separating the output-capture error from the original guard failure without exposing sensitive logs. Could you help us assess:
Thank you again for helping us narrow this down. |
Uh oh!
There was an error while loading. Please reload this page.
🏷️ Discussion Type
Question
💬 Feature/Topic Area
Workflow Deployment
Discussion Details
Hello GitHub Community,
We deploy a self-hosted business application using GitHub Actions, appleboy/ssh-action, and Docker Compose.
The existing website remains accessible, but a deployment failed with these messages, in this order:
Code-only guard failed; stop and review. No full-deploy fallback.
Staged deployment incomplete
Operation incomplete; persistent interlock retained
A subsequent error was:
Invalid value. Matching delimiter not found 'EOF'
Important context:
Action fails to set stdout when one of the commands has non 0 exit code appleboy/ssh-action#375
We would appreciate advice on:
Could the EOF message be a secondary output-capture error after the original remote command failure? How should the workflow preserve the original error and exit status reliably?
What minimal, credential-free reproduction would help distinguish an ssh-action output issue from a failure in our custom deployment script?
How should we design and test recovery for a partially completed Docker Compose deployment, preserving data and recording the outcome before allowing another deployment?
We are seeking source-level review and isolated testing guidance. We do not want to remove the persistent marker blindly, weaken compatibility checks, or force a production deployment.
We can provide sanitized workflow excerpts and a minimal reproduction if needed. No credentials or production business data will be shared.
Thank you.
All reactions