Failed deployment stays recorded as deployed, so it is not retried and not reported again

[denisolnce]

What happens

A stack of mine has a post_start hook that runs database migrations. The hook
exited 1, so the deployment failed, correctly:

end stage early: deploy  reason: failed to deploy stack app: failed to deploy
stack: login-be hook exited with status 1
deployment failed  stack=app target=login-dev

The next poll, 30 seconds later, said:

comparing commits  deployed_commit=573a16e  latest_commit=573a16e
no changes detected, skipping deployment  stack=app

So the failed commit counts as deployed. The containers run the new images, the
migrations never ran, and the failure was reported exactly once. Nothing retried
it, nothing reminded me. If I had not watched the log, only one message in the
chat would tell that the environment is half deployed.

Why

The deployment state lives on the containers, and it is written when compose
creates them:

• addComposeLabels (internal/docker/compose.go) puts
  DocoCDLabels.Deployment.CommitSHA = latest commit into every service's custom
  labels, before up runs.
• Pre-deploy reads it back with deployedState.GetDeploymentCommitSHA()
  (internal/stages/stage_2_pre-deploy.go) and skips when it equals the latest
  commit.

A post_start hook runs after the container started, so the label with the new
commit already exists when the hook fails. Same for anything else that fails after
containers are created.

The behaviour is asymmetric, which makes it easy to miss:

• Pull failure - no container is created, labels stay old, so every poll retries
  and reports. That is loud, sometimes too loud.
• Failure after creation - state says deployed, so no retry and no second report.
  That is silent, and it is the dangerous one.

Proposal

Record the outcome, not only the commit. Two shapes:

1. Keep the labels as they are and add a deployment outcome to the state, for
   example a doco-cd.deployment.status label written after the stages finish.
   Pre-deploy then treats "same commit, last outcome failed" as a change and
   deploys again. Retry keeps working like the pull-failure case, and with the
   de-duplication from fix(notification): stop repeating and duplicating failure notifications #1700 it costs one message plus reminders, not a flood.
2. Stamp the commit label only after a successful deploy. Cleaner in theory, but
   labels are part of the project model that compose applies during up, so this
   needs a second pass over the containers and I like it less.

I would go with option 1.

Two questions before I open a PR:

• Do you want the retry to be endless, like the pull-failure path, or bounded?
• Is a new label acceptable, or do you prefer the outcome in the daemon's data
  volume instead? A label survives a daemon restart, which is why I picked it.

[kimdre]

I would also go with Option 1 with endless retries until it succeeds and a new label. I always try to use the container labels to store any data that needs to survive a doco-cd restart.

Maybe we can think about a opt-in rollback functionality for deployments sometime later for cases where a succesful deployment of the new state is not possible.

[denisolnce]

Opened #1711. One deviation I have to confess: the outcome is a marker file in the data volume, not a label. Turned out labels cannot be written to existing containers at all - compose stamps them only at creation, and docker has no API to change them later. So a post-stages label write needs exactly the recreate pass that made us both dislike option 2. Marker survives daemon restarts same as labels do, is removed on first success, and the retry force-recreates the stack so a failed hook actually runs again instead of no-op'ing into a false success. Endless retries, one dedup'd notification per fault as discussed.