Test bootstrap restore in CI #2061

alexandre-allard · 2019-11-21T09:20:07Z

Component: ci, tests

Context:

Summary:

Acceptance criteria: Green build

Closes: #1687

bert-e · 2019-11-21T09:20:36Z

Hello alexandre-allard-scality,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

bert-e · 2019-11-21T09:20:40Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

bert-e · 2019-11-22T16:21:18Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/1687-test-restore-in-ci into
development/2.4
w/2.5/improvement/1687-test-restore-in-ci will be merged into development/2.5

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

bert-e · 2019-11-22T16:21:20Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

bert-e · 2019-11-24T17:38:42Z

History mismatch

Merge commit #68424c5580f49f92de6154607f5b4a30979a6e92 on the integration branch
w/2.5/improvement/1687-test-restore-in-ci is merging a branch which is neither the current
branch improvement/1687-test-restore-in-ci nor the development branch
development/2.5.

It is likely due to a rebase of the branch improvement/1687-test-restore-in-ci and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

bert-e · 2019-11-27T10:05:06Z

Branches have diverged

This pull request's source branch improvement/1687-test-restore-in-ci has diverged from
development/2.4 by more than 50 commits.

To avoid any integration risks, please re-synchronize them using one of the
following solutions:

Merge origin/development/2.4 into improvement/1687-test-restore-in-ci
Rebase improvement/1687-test-restore-in-ci onto origin/development/2.4

Note: If you choose to rebase, you may have to ask me to rebuild
integration branches using the reset command.

bert-e · 2019-11-27T17:00:45Z

History mismatch

Merge commit #2498e576cf484803cfbd102f5fae9752f54302e8 on the integration branch
w/2.5/improvement/1687-test-restore-in-ci is merging a branch which is neither the current
branch improvement/1687-test-restore-in-ci nor the development branch
development/2.5.

It is likely due to a rebase of the branch improvement/1687-test-restore-in-ci and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

gdemonet · 2019-11-27T23:23:46Z

Great effort, seems like you encountered much more problems than anticipated. However, I think we don't want to keep adding things to Eve specifics, but instead extend the work that was done in tests/install for cluster expansion, and/or move more logic into Terraform and make this accessible to users. Anyway, going from this to something more generic shouldn't be too hard now that you solved most issues.

eve/main.yml

TeddyAndrieux

Small comment but LGTM

alexandre-allard · 2020-02-11T14:10:02Z

/reset

TeddyAndrieux · 2020-05-20T15:18:45Z

/reset

bert-e · 2020-05-20T15:18:54Z

Reset complete

I have successfully deleted this pull request's integration branches.

bert-e · 2020-05-20T15:19:01Z

Conflict

A conflict has been raised during the creation of
integration branch w/2.5/improvement/1687-test-restore-in-ci with contents from improvement/1687-test-restore-in-ci
and development/2.5.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/2.5/improvement/1687-test-restore-in-ci origin/development/2.5
 $ git merge origin/improvement/1687-test-restore-in-ci
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/2.5/improvement/1687-test-restore-in-ci

bert-e · 2020-05-20T16:25:44Z

Conflict

A conflict has been raised during the creation of
integration branch w/2.6/improvement/1687-test-restore-in-ci with contents from w/2.5/improvement/1687-test-restore-in-ci
and development/2.6.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/2.6/improvement/1687-test-restore-in-ci origin/development/2.6
 $ git merge origin/w/2.5/improvement/1687-test-restore-in-ci
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/2.6/improvement/1687-test-restore-in-ci

bert-e · 2020-05-25T07:19:02Z

Conflict

A conflict has been raised during the update of
integration branch w/2.5/improvement/1687-test-restore-in-ci with contents from improvement/1687-test-restore-in-ci
and development/2.5.

Please resolve the conflict on the integration branch (w/2.5/improvement/1687-test-restore-in-ci).

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout w/2.5/improvement/1687-test-restore-in-ci
 $ git pull  # or "git reset --hard origin/w/2.5/improvement/1687-test-restore-in-ci"
 $ git merge origin/development/2.5
 $ # <intense conflict resolution>
 $ git commit
 $ git merge origin/improvement/1687-test-restore-in-ci
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/2.5/improvement/1687-test-restore-in-ci

Apiserver proxy redirect to an old bootstrap node that may not longer exists, so lets reconfigure apiserver-proxy on every nodes to not longer talk to the old bootstrap node

Deploy two node instead of only one during expansion tests to have HA etcd and being able to test the bootstrap restoration Refs: #1687

Refs: #1687

(cherry picked from commit 1e110a3)

This `then` will be useful in restoration tests also so we move it to a common place Refs: #1687

Adding `EXTRA_OPTS` environment variable in `wait_pods_status` helper to allow to add extra arguments like "--namespace" if needed

We now create AlertManager & Prometheus volumes on node1 instead of the bootstrap node on multi node environment to properly test that storage volume does not need to sit on bootstrap node

We need to be able to connect to all the nodes to do the expansion from the bastion host, create an helper in eve to do it

we don't want to check presence of kube-dns pods on the bootstrap node, because pods can move to other node, especially during restore tests Refs: #1687

This scenario run the restore script and ensure that everything is working as expected (pods, ...) after the restoration. Refs: #1687

Launch bootstrap restore tests in CI during post-merge stages, this CI step first need a full MetalK8s cluster with at least 3-node etcd cluster Refs: #1687

bert-e · 2020-05-25T08:30:48Z

History mismatch

Merge commit #c6a3bee8fde5180bfefdf29af4869211091ac97d on the integration branch
w/2.5/improvement/1687-test-restore-in-ci is merging a branch which is neither the current
branch improvement/1687-test-restore-in-ci nor the development branch
development/2.5.

It is likely due to a rebase of the branch improvement/1687-test-restore-in-ci and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

bert-e · 2020-05-25T09:10:34Z

Conflict

A conflict has been raised during the creation of
integration branch w/2.6/improvement/1687-test-restore-in-ci with contents from w/2.5/improvement/1687-test-restore-in-ci
and development/2.6.

I have not created the integration branch.

Here are the steps to resolve this conflict:

 $ git fetch
 $ git checkout -B w/2.6/improvement/1687-test-restore-in-ci origin/development/2.6
 $ git merge origin/w/2.5/improvement/1687-test-restore-in-ci
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/2.6/improvement/1687-test-restore-in-ci

bert-e · 2020-05-28T09:18:55Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

gdemonet

Nothing preventing the merge, but Terraform, our tests, and Eve's YAML need to be cleaned-up and refactored. For the future :)

gdemonet · 2020-06-02T06:51:18Z

eve/main.yml

@@ -1597,7 +1597,7 @@ stages:
          name: Run installation scenarii on the bastion
          env:
            <<: *_env_bastion_tests
-            PYTEST_FILTERS: "install and ci and multinodes"
+            PYTEST_FILTERS: "install and ci and multinodes and not node2"


Beurk. Let's keep it, but really not what BDD tags should be used for, IMHO

Yes but we do not have better solution here for expansion right now I think

gdemonet · 2020-06-02T06:58:39Z

eve/main.yml

+  - ShellCommand: &copy_bastion_pub_key_ssh
+      name: Send bastion public key to nodes
+      command: >
+        for host in $HOSTS_LIST; do
+          ssh -F ssh_config bastion "cat .ssh/bastion.pub" |
+          ssh -F ssh_config $host "cat >> .ssh/authorized_keys"
+        done
+      env:
+        HOSTS_LIST: bootstrap
+      workdir: build/eve/workers/openstack-multiple-nodes/terraform/
+


Why not do this in Terraform?

It was not done before but yes maybe this could go in terraform

eve/main.yml

gdemonet · 2020-06-02T07:04:43Z

eve/main.yml

+      - SetPropertyFromCommand:
+          name: Set node1 etcd container id
+          property: node1_etcd_container_id
+          command: >
+            ssh -F ssh_config node1
+            sudo crictl ps -q --label io.kubernetes.pod.namespace=kube-system
+            --label io.kubernetes.container.name=etcd --state Running
+          workdir: build/eve/workers/openstack-multiple-nodes/terraform/
+          haltOnFailure: true
+      - SetPropertyFromCommand:
+          name: Set boostrap node etcd member id
+          property: bootstrap_etcd_member_id
+          command: >
+            ssh -F ssh_config node1
+            sudo crictl exec -i "%(prop:node1_etcd_container_id)s" sh -c \"
+            ETCDCTL_API=3 etcdctl --endpoints https://127.0.0.1:2379
+            --cert /etc/kubernetes/pki/etcd/server.crt
+            --key /etc/kubernetes/pki/etcd/server.key
+            --cacert /etc/kubernetes/pki/etcd/ca.crt
+            member list\" | awk -F ', ' '$3 ~ "bootstrap" { print $1 }'
+          workdir: build/eve/workers/openstack-multiple-nodes/terraform/
+          haltOnFailure: true
+      - ShellCommand:
+          name: Remove bootstrap node from etcd members
+          command: >
+            ssh -F ssh_config node1
+            sudo crictl exec -i "%(prop:node1_etcd_container_id)s" sh -c \"
+            ETCDCTL_API=3 etcdctl --endpoints https://127.0.0.1:2379
+            --cert /etc/kubernetes/pki/etcd/server.crt
+            --key /etc/kubernetes/pki/etcd/server.key
+            --cacert /etc/kubernetes/pki/etcd/ca.crt
+            member remove %(prop:bootstrap_etcd_member_id)s\"
+          workdir: build/eve/workers/openstack-multiple-nodes/terraform/
+          haltOnFailure: true
+      - ShellCommand:
+          name: Remove bootstrap node object
+          command: >
+            ssh -F ssh_config node1
+            sudo kubectl --kubeconfig=/etc/kubernetes/admin.conf delete
+            node --selector="node-role.kubernetes.io/bootstrap"
+          workdir: build/eve/workers/openstack-multiple-nodes/terraform/
+          haltOnFailure: true


Why isn't all this orchestrated in the test? Looks like it's not in "the right place"...

Yes could be part of the pytest test directly but then we need to do it after spawning the new bootstrap node, anyway yes it's doable

alexandre-allard · 2020-06-02T08:03:58Z

/approve

bert-e · 2020-06-02T08:04:14Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/2.4
✔️ development/2.5
✔️ development/2.6

The following branches will NOT be impacted:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2020-06-02T12:18:19Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.4
✔️ development/2.5
✔️ development/2.6

The following branches have NOT changed:

development/1.0
development/1.1
development/1.2
development/1.3
development/2.0
development/2.1
development/2.2
development/2.3

Please check the status of the associated issue None.

Goodbye alexandre-allard-scality.

alexandre-allard force-pushed the improvement/1687-test-restore-in-ci branch 3 times, most recently from 2bec115 to 94965b3 Compare November 22, 2019 16:21

alexandre-allard force-pushed the improvement/1687-test-restore-in-ci branch from 94965b3 to 04b2bb2 Compare November 24, 2019 17:38

alexandre-allard force-pushed the improvement/1687-test-restore-in-ci branch 2 times, most recently from e101352 to 3cada92 Compare November 25, 2019 08:06

alexandre-allard force-pushed the improvement/1687-test-restore-in-ci branch from 3cada92 to 2316a62 Compare November 27, 2019 17:00

alexandre-allard force-pushed the improvement/1687-test-restore-in-ci branch 8 times, most recently from 09ffe18 to c0fd193 Compare December 3, 2019 09:29

alexandre-allard marked this pull request as ready for review December 3, 2019 09:29

alexandre-allard requested a review from a team as a code owner December 3, 2019 09:29

alexandre-allard force-pushed the improvement/1687-test-restore-in-ci branch from c0fd193 to 8e18864 Compare December 3, 2019 10:49

alexandre-allard force-pushed the improvement/1687-test-restore-in-ci branch from 8e18864 to ccb5142 Compare February 11, 2020 07:47

TeddyAndrieux reviewed Feb 11, 2020

View reviewed changes

eve/main.yml Outdated Show resolved Hide resolved

TeddyAndrieux previously approved these changes Feb 11, 2020

View reviewed changes

TeddyAndrieux and others added 11 commits May 25, 2020 10:30

scripts: Re-configure apiserver-proxy on all nodes during restore

273a1b6

Apiserver proxy redirect to an old bootstrap node that may not longer exists, so lets reconfigure apiserver-proxy on every nodes to not longer talk to the old bootstrap node

tests/ci: add a second node to expansion tests if node2

774ce0f

Deploy two node instead of only one during expansion tests to have HA etcd and being able to test the bootstrap restoration Refs: #1687

tests/ci: pass backup archive env var to pytest

2177dc4

Refs: #1687

tests: add a get_grain helper and use it everywhere

9155b13

(cherry picked from commit 1e110a3)

tests: move check_etcd_role definition to conftest

e194582

This `then` will be useful in restoration tests also so we move it to a common place Refs: #1687

ci: Add extra_opts for wait running Pods

5740ee8

Adding `EXTRA_OPTS` environment variable in `wait_pods_status` helper to allow to add extra arguments like "--namespace" if needed

ci: create volumes on node1 instead of bootstrap for multi-node

be18d3e

We now create AlertManager & Prometheus volumes on node1 instead of the bootstrap node on multi node environment to properly test that storage volume does not need to sit on bootstrap node

ci: copy bastion pub key to authorized keys to a list of nodes

d97fc25

We need to be able to connect to all the nodes to do the expansion from the bastion host, create an helper in eve to do it

tests: no longer check kube-dns node

ca8b522

we don't want to check presence of kube-dns pods on the bootstrap node, because pods can move to other node, especially during restore tests Refs: #1687

tests/ci: add tests scenario for bootstrap restoration

4b7a967

This scenario run the restore script and ensure that everything is working as expected (pods, ...) after the restoration. Refs: #1687

ci: add bootstrap restore tests in CI

0293ddd

Launch bootstrap restore tests in CI during post-merge stages, this CI step first need a full MetalK8s cluster with at least 3-node etcd cluster Refs: #1687

TeddyAndrieux force-pushed the improvement/1687-test-restore-in-ci branch from bd08570 to 0293ddd Compare May 25, 2020 08:30

gdemonet approved these changes Jun 2, 2020

View reviewed changes

TeddyAndrieux approved these changes Jun 2, 2020

View reviewed changes

bert-e merged commit 47c3c09 into development/2.4 Jun 2, 2020

bert-e deleted the improvement/1687-test-restore-in-ci branch June 2, 2020 12:18

This was referenced Sep 15, 2020

Retrieve the backup/restore archive from a bootstrap node #1805

Closed

Spawn a new bootstrap node using the backup archive #1807

Closed

Destroy the bootstrap node in an attempt to test backup and restoration. #1806

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test bootstrap restore in CI #2061

Test bootstrap restore in CI #2061

alexandre-allard commented Nov 21, 2019

bert-e commented Nov 21, 2019

bert-e commented Nov 21, 2019

bert-e commented Nov 22, 2019

bert-e commented Nov 22, 2019

bert-e commented Nov 24, 2019

bert-e commented Nov 27, 2019

bert-e commented Nov 27, 2019

gdemonet commented Nov 27, 2019

TeddyAndrieux left a comment

alexandre-allard commented Feb 11, 2020

TeddyAndrieux commented May 20, 2020

bert-e commented May 20, 2020

bert-e commented May 20, 2020

bert-e commented May 20, 2020

bert-e commented May 25, 2020

bert-e commented May 25, 2020

bert-e commented May 25, 2020

bert-e commented May 28, 2020

gdemonet left a comment

gdemonet Jun 2, 2020

TeddyAndrieux Jun 2, 2020

gdemonet Jun 2, 2020

TeddyAndrieux Jun 2, 2020

gdemonet Jun 2, 2020

TeddyAndrieux Jun 2, 2020

alexandre-allard commented Jun 2, 2020

bert-e commented Jun 2, 2020

bert-e commented Jun 2, 2020

Test bootstrap restore in CI #2061

Test bootstrap restore in CI #2061

Conversation

alexandre-allard commented Nov 21, 2019

bert-e commented Nov 21, 2019

Hello alexandre-allard-scality,

bert-e commented Nov 21, 2019

Waiting for approval

bert-e commented Nov 22, 2019

Integration data created

bert-e commented Nov 22, 2019

Waiting for approval

bert-e commented Nov 24, 2019

History mismatch

bert-e commented Nov 27, 2019

Branches have diverged

bert-e commented Nov 27, 2019

History mismatch

gdemonet commented Nov 27, 2019

TeddyAndrieux left a comment

Choose a reason for hiding this comment

alexandre-allard commented Feb 11, 2020

TeddyAndrieux commented May 20, 2020

bert-e commented May 20, 2020

Reset complete

bert-e commented May 20, 2020

Conflict

bert-e commented May 20, 2020

Conflict

bert-e commented May 25, 2020

Conflict

bert-e commented May 25, 2020

History mismatch

bert-e commented May 25, 2020

Conflict

bert-e commented May 28, 2020

Waiting for approval

gdemonet left a comment

Choose a reason for hiding this comment

gdemonet Jun 2, 2020

Choose a reason for hiding this comment

TeddyAndrieux Jun 2, 2020

Choose a reason for hiding this comment

gdemonet Jun 2, 2020

Choose a reason for hiding this comment

TeddyAndrieux Jun 2, 2020

Choose a reason for hiding this comment

gdemonet Jun 2, 2020

Choose a reason for hiding this comment

TeddyAndrieux Jun 2, 2020

Choose a reason for hiding this comment

alexandre-allard commented Jun 2, 2020

bert-e commented Jun 2, 2020

In the queue

bert-e commented Jun 2, 2020