increment calculation on release can hide failures

[wouterdb]

Problem

1. a version v1 is deploying
2. a new version v2 is released
3. resource a[k=x],v=2 is marked as deployed for known good state
4. resource a[k=x],v=1 fails

Now, the good state of v2 has masked the bad state of v1.

Current solution proposal

1. when a deploy fails
2. on a version that is not the latest
3. and the corresponding resource on the latest version is set to deploy
   • this can only be due to increment calculation, because the agent can only start deploying the latest released version at any time, so the newer version has never been deployed by the agent, so is was set by the increment. We should be able to assert that the attribute has MUST be identical
4. then we set it to failed instead. (with a message: setting to failed because of failure in previous version xyz)

The first two conditions are fairly cheap to check

Transactional behavior

• The idea would be to execute this code, in a separate transaction, at the end of the 'deploy_done' method
• However, it races with the actual release_version (and other full) increment calculation
• We can however retry this write until we catch up
• question: is it sufficient to make this a retryable serializable transaction to make it retry on this