WIP: Split deploy_node and prepare etcd members for cluster addition #2147

Ebaneck · 2019-12-18T08:08:25Z

Component:

'salt'

Context:

See #2028

Summary:

Split deploy_node into generic stages
Add prepared state to prepare etcd deps and register etcd node into etcd cluster
Add running state to create manifest and wait for member to be healthy

Acceptance criteria:

Closes: #2028

bert-e · 2019-12-18T08:08:28Z

Hello ebaneck,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

bert-e · 2019-12-18T08:08:34Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge bugfix/2028-ensure-etcd-is-running-before-registering-members into
development/2.4
w/2.5/bugfix/2028-ensure-etcd-is-running-before-registering-members will be merged into development/2.5

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

bert-e · 2019-12-18T08:08:36Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

bert-e · 2019-12-18T08:52:30Z

History mismatch

Merge commit #a62df6e25135fc7693e0c4e2cdd06721de67d45c on the integration branch
w/2.5/bugfix/2028-ensure-etcd-is-running-before-registering-members is merging a branch which is neither the current
branch bugfix/2028-ensure-etcd-is-running-before-registering-members nor the development branch
development/2.5.

It is likely due to a rebase of the branch bugfix/2028-ensure-etcd-is-running-before-registering-members and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

salt/metalk8s/kubernetes/etcd/installed.sls

salt/metalk8s/orchestrate/deploy_node.sls

salt/metalk8s/roles/etcd/prepared.sls

salt/metalk8s/roles/etcd/running.sls

salt/metalk8s/orchestrate/deploy_node.sls

Add a small delay right after `apiserver-proxy` manifest get changes so that we didn't check the "old" apiserver-proxy

If we try to deploy `apiserver-proxy` on an "master" node we will have `weight=100` for the local apiserver but this local apiserver may not be ready or configured yet, so we need to be able to skip the `apiserver-proxy` healthcheck from the salt state

When we run `etcd` state we want to check `etcd` health by default but when we deploy a new `etcd` the new member is not yet registered in the `etcd` cluster so we need to skip the healthcheck of the `etcd` when deploying this new `etcd` node, so adding a pillar value to skip `etcd` healthcheck in the salt state

This commit splits node deployment procedure into a sequence of stages. Initially, expanding the etcd cluster from 1 to 2 nodes puts the cluster in a blocked state. This happens when a new member is declared and not yet started. If this time to start is too long, liveness probes on the initial etcd Pod will fail, then it will be restarted and then enter a back-off loop. Even after the new etcd Pod has been created, the new one won't be able to start, since the initial is in back-off and will enter a back-off as well. To prevent the above situation, we pre-pull the etcd image on the target Node, which ensures the shortest time possible between the new member being registered and its corresponding Pod being created. Note that this reordering is inspired by kubespray

TeddyAndrieux · 2019-12-23T16:51:27Z

/reset

bert-e · 2019-12-23T16:51:34Z

Reset complete

I have successfully deleted this pull request's integration branches.

bert-e · 2019-12-23T16:51:42Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge bugfix/2028-ensure-etcd-is-running-before-registering-members into
development/2.4
w/2.5/bugfix/2028-ensure-etcd-is-running-before-registering-members will be merged into development/2.5

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

bert-e · 2019-12-23T16:51:44Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

bert-e · 2019-12-23T18:08:36Z

History mismatch

Merge commit #edb8a3da12b20f4a2242b2af1d109192f41be898 on the integration branch
w/2.5/bugfix/2028-ensure-etcd-is-running-before-registering-members is merging a branch which is neither the current
branch bugfix/2028-ensure-etcd-is-running-before-registering-members nor the development branch
development/2.5.

It is likely due to a rebase of the branch bugfix/2028-ensure-etcd-is-running-before-registering-members and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

During deploy node we restart some mandatory pod (salt-master, repositories) at several place so we need to check pillar value availability several time

TeddyAndrieux · 2019-12-23T18:09:41Z

/reset

bert-e · 2019-12-23T18:09:49Z

Reset complete

I have successfully deleted this pull request's integration branches.

bert-e · 2019-12-23T18:09:57Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge bugfix/2028-ensure-etcd-is-running-before-registering-members into
development/2.4
w/2.5/bugfix/2028-ensure-etcd-is-running-before-registering-members will be merged into development/2.5

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

bert-e · 2019-12-23T18:09:58Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

bert-e · 2019-12-23T18:53:26Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

@NicolasT

Ebaneck · 2019-12-23T20:51:23Z

/approve

bert-e · 2019-12-23T20:51:29Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

@NicolasT

The following options are set: approve

stale

bert-e · 2019-12-23T20:56:06Z

Build failed

The build for commit did not succeed in branch w/2.5/bugfix/2028-ensure-etcd-is-running-before-registering-members.

The following options are set: approve

bert-e · 2019-12-23T22:00:03Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/2.4
✔️ development/2.5

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2019-12-23T22:00:08Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.4
✔️ development/2.5

The following branches have NOT changed:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3

Please check the status of the associated issue None.

Goodbye ebaneck.

Ebaneck force-pushed the bugfix/2028-ensure-etcd-is-running-before-registering-members branch from 17251b1 to bc7630e Compare December 18, 2019 08:52

Ebaneck force-pushed the bugfix/2028-ensure-etcd-is-running-before-registering-members branch 6 times, most recently from 933a12a to db55c68 Compare December 18, 2019 11:55

TeddyAndrieux requested changes Dec 18, 2019

View reviewed changes

TeddyAndrieux reviewed Dec 18, 2019

View reviewed changes

salt/metalk8s/orchestrate/deploy_node.sls Outdated Show resolved Hide resolved

Ebaneck force-pushed the bugfix/2028-ensure-etcd-is-running-before-registering-members branch 3 times, most recently from 7b85a6f to 3027628 Compare December 18, 2019 17:47

Ebaneck requested a review from TeddyAndrieux December 18, 2019 17:54

TeddyAndrieux reviewed Dec 18, 2019

View reviewed changes

salt/metalk8s/orchestrate/deploy_node.sls Outdated Show resolved Hide resolved

Ebaneck force-pushed the bugfix/2028-ensure-etcd-is-running-before-registering-members branch 3 times, most recently from 6d8cab5 to c3336e4 Compare December 18, 2019 18:42

Ebaneck requested a review from TeddyAndrieux December 18, 2019 18:42

Ebaneck marked this pull request as ready for review December 18, 2019 18:42

Ebaneck requested a review from a team as a code owner December 18, 2019 18:42

Ebaneck force-pushed the bugfix/2028-ensure-etcd-is-running-before-registering-members branch 6 times, most recently from 0befc0e to 22f04ec Compare December 19, 2019 10:18

TeddyAndrieux force-pushed the bugfix/2028-ensure-etcd-is-running-before-registering-members branch 3 times, most recently from 6c251fb to 2aa4cac Compare December 23, 2019 12:18

TeddyAndrieux added 3 commits December 23, 2019 13:19

salt: Add delay after apiserver-proxy manifest changes

7e3abfa

Add a small delay right after `apiserver-proxy` manifest get changes so that we didn't check the "old" apiserver-proxy

TeddyAndrieux force-pushed the bugfix/2028-ensure-etcd-is-running-before-registering-members branch 2 times, most recently from 6ca46a2 to edb8a3d Compare December 23, 2019 16:26

TeddyAndrieux force-pushed the bugfix/2028-ensure-etcd-is-running-before-registering-members branch from edb8a3d to f528911 Compare December 23, 2019 18:08

salt: Add some refresh and check pillar in deploy node

f528911

During deploy node we restart some mandatory pod (salt-master, repositories) at several place so we need to check pillar value availability several time

TeddyAndrieux approved these changes Dec 23, 2019

View reviewed changes

bert-e merged commit f528911 into development/2.4 Dec 23, 2019

bert-e deleted the bugfix/2028-ensure-etcd-is-running-before-registering-members branch December 23, 2019 22:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Split deploy_node and prepare etcd members for cluster addition #2147

WIP: Split deploy_node and prepare etcd members for cluster addition #2147

Ebaneck commented Dec 18, 2019

bert-e commented Dec 18, 2019

bert-e commented Dec 18, 2019

bert-e commented Dec 18, 2019

bert-e commented Dec 18, 2019

TeddyAndrieux commented Dec 23, 2019

bert-e commented Dec 23, 2019

bert-e commented Dec 23, 2019

bert-e commented Dec 23, 2019

bert-e commented Dec 23, 2019

TeddyAndrieux commented Dec 23, 2019

bert-e commented Dec 23, 2019

bert-e commented Dec 23, 2019

bert-e commented Dec 23, 2019

bert-e commented Dec 23, 2019

Ebaneck commented Dec 23, 2019

bert-e commented Dec 23, 2019

bert-e commented Dec 23, 2019

bert-e commented Dec 23, 2019

bert-e commented Dec 23, 2019

WIP: Split deploy_node and prepare etcd members for cluster addition #2147

WIP: Split deploy_node and prepare etcd members for cluster addition #2147

Conversation

Ebaneck commented Dec 18, 2019

bert-e commented Dec 18, 2019

Hello ebaneck,

bert-e commented Dec 18, 2019

Integration data created

bert-e commented Dec 18, 2019

Waiting for approval

bert-e commented Dec 18, 2019

History mismatch

TeddyAndrieux commented Dec 23, 2019

bert-e commented Dec 23, 2019

Reset complete

bert-e commented Dec 23, 2019

Integration data created

bert-e commented Dec 23, 2019

Waiting for approval

bert-e commented Dec 23, 2019

History mismatch

TeddyAndrieux commented Dec 23, 2019

bert-e commented Dec 23, 2019

Reset complete

bert-e commented Dec 23, 2019

Integration data created

bert-e commented Dec 23, 2019

Waiting for approval

bert-e commented Dec 23, 2019

Waiting for approval

Ebaneck commented Dec 23, 2019

bert-e commented Dec 23, 2019

Waiting for approval

bert-e commented Dec 23, 2019

Build failed

bert-e commented Dec 23, 2019

In the queue

bert-e commented Dec 23, 2019