salt/drain: Handle retryable errors when evicting

[gdemonet]

The Eviction API sends '429 Too Many Requests' when a requested eviction
can't be applied due to some disruption budget. This way, the client can
wait and retry later.
We didn't handle this error in our implementation, hence rolling
upgrades were failing as soon as we hit this situation.
We add support for this, and also move the "timeout" scope to the whole
eviction process (since we now may be stuck at the eviction creation).

Fixes: #2530

---

Closes: #2530

[bert-e]

Hello gdemonet,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

[bert-e]

Integration data created

I have created the integration data for the additional destination branches.

• this pull request will merge bugfix/2530-drain-wait into
  development/2.0
• w/2.1/bugfix/2530-drain-wait will be merged into development/2.1
• w/2.2/bugfix/2530-drain-wait will be merged into development/2.2
• w/2.3/bugfix/2530-drain-wait will be merged into development/2.3
• w/2.4/bugfix/2530-drain-wait will be merged into development/2.4
• w/2.5/bugfix/2530-drain-wait will be merged into development/2.5
• w/2.6/bugfix/2530-drain-wait will be merged into development/2.6

The following branches will NOT be impacted:

• development/1.0
• development/1.1
• development/1.2
• development/1.3

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• one peer

Peer approvals must include at least 1 approval from the following list:

• @nootal

• @alexandre-allard-scality

• @lucieleonard

• @sayf-eddine-scality

• @slaperche-scality

• @NicolasT

• @gdemonet

• @carlito-scality

• @thomasdanan

• @jbertran

• @TeddyAndrieux

• @cmonfort

• @MonPote

[gdemonet]

Tested with a 3 workers deployment of Zenko, didn't get any issue with the drain during an upgrade!

[gdemonet]

/approve

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• one peer

Peer approvals must include at least 1 approval from the following list:

• @nootal

• @alexandre-allard-scality

• @lucieleonard

• @sayf-eddine-scality

• @slaperche-scality

• @NicolasT

• @gdemonet

• @carlito-scality

• @thomasdanan

• @jbertran

• @TeddyAndrieux

• @cmonfort

• @MonPote

The following options are set: approve

[TeddyAndrieux]

I would have liked more debug log but LGTM

[bert-e]

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

• ✔️ development/2.0

• ✔️ development/2.1

• ✔️ development/2.2

• ✔️ development/2.3

• ✔️ development/2.4

• ✔️ development/2.5

• ✔️ development/2.6

The following branches will NOT be impacted:

• development/1.0
• development/1.1
• development/1.2
• development/1.3

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

• Any commit you add on the source branch will trigger a new cycle after the
  current queue is merged.
• Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

[gdemonet]

I would have liked more debug log but LGTM

You're right, I'll open a follow-up PR.

[bert-e]

I have successfully merged the changeset of this pull request
into targetted development branches:

• ✔️ development/2.0

• ✔️ development/2.1

• ✔️ development/2.2

• ✔️ development/2.3

• ✔️ development/2.4

• ✔️ development/2.5

• ✔️ development/2.6

The following branches have NOT changed:

• development/1.0
• development/1.1
• development/1.2
• development/1.3

Please check the status of the associated issue None.

Goodbye gdemonet.