bin/upgrade: handle conflict errors

[jone]

When running upgrade steps with bin/upgrade install -Sp and a ConflictError happens at the end, we have these problems:

1. the conflict error happens in Zope after the view is terminated an the request logger is terminated, which results in a terminated response and the bin/upgrade command quits with an error although the zope makes a "retry".

2. the "retry" can never work, because the API tries to validate the authentication token using the authentication tempfile, but it was deleted be the first try, leading to an error.

It would be awesome if bin/upgrade would support conflict errors / retries. If we want to go crazy we could even have an option for changing the maximum amount of retries (which is 2 to by default).

[deiferni]

This has surfaced again today with slightly different behavior. After a ConflictError the retry was executed successfully without the token validation error mentioned above. If you're not really observing the upgrade output all the time you might easily be tricked into thinking that the upgrade just takes very long and miss the fact that all the upgrades are just rerun. Additionally when a ConflictError occurs once in the upgrades it is not unlikely it will happen again in the retry. I would welcome an option to prevent retries during upgrades, this could be achieved by the option mentioned in this issue to set the maximum amount of retries.
We need to pick up this issue in the near future and discuss possible approaches and a good default for retries during upgrades.