Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add backup/restore test in the CI #1687

Closed
TeddyAndrieux opened this issue Sep 13, 2019 · 0 comments · Fixed by #2061
Closed

Add backup/restore test in the CI #1687

TeddyAndrieux opened this issue Sep 13, 2019 · 0 comments · Fixed by #2061
Assignees
@alexandre-allard @TeddyAndrieux
Labels
complexity:medium Something that requires one or few days to fix topic:tests What's not tested may be broken

Comments

@TeddyAndrieux
Copy link
Collaborator

Component:

'tests'

Why this is needed:

Currently backup is launch at the end of each bootstrap but restore is never tested

What should be done:

Add a test for bootstrap recovery

Implementation proposal (strongly recommended):

Spawn a metalk8s cluster retrieve the backup file from/var/lib/metalk8s/ destroy the bootstrap node, spawn a new machine and use it to spawn a new bootstrap node
@TeddyAndrieux TeddyAndrieux added topic:tests What's not tested may be broken moonshot complexity:medium Something that requires one or few days to fix labels Sep 13, 2019
@thomasdanan thomasdanan added this to the MetalK8s 2.4.0-beta1 milestone Sep 16, 2019
@thomasdanan thomasdanan removed this from the MetalK8s 2.4.0-beta1 milestone Sep 16, 2019
@alexandre-allard alexandre-allard self-assigned this Nov 14, 2019
alexandre-allard added a commit that referenced this issue Nov 21, 2019
@alexandre-allard
ci: Add jq as a requirements for multiple-nodes
2498e57 
We need jq to parse json output from some commands
in the CI context in multiple-nodes tests

Refs: #1687
alexandre-allard added a commit that referenced this issue Nov 21, 2019
@alexandre-allard
ci: create control plane network & interfaces
ba49690 
Creation of the network used by the control plane,
this allows the usage of a VIP for apiserver which
will be used to test backup/restoration mechanism

Refs: #1687
alexandre-allard added a commit that referenced this issue Nov 21, 2019
@alexandre-allard
ci: add the control plane port to the nodes
043ef3a 
Link the control plane interfaces to the nodes

Refs: #1687
alexandre-allard added a commit that referenced this issue Nov 21, 2019
@alexandre-allard
ci: add script to configure control plane iface
674f91e 
This script creates the configuration of the new
network interface used by the control plane network
and ensures that this interface is up.
It would be better to replace it by a cloud-init
script at some point, but for now, it does the job.

Refs: #1687
alexandre-allard added a commit that referenced this issue Nov 21, 2019
@alexandre-allard
ci: use control plane network & vip in bootstrap config
6cc1323 
Use the dedicated control plane network and VIP
in the bootstrap configuration script.
Also enable the usage of keepalived.

Refs: #1687
alexandre-allard added a commit that referenced this issue Nov 21, 2019
@alexandre-allard
tests: fix salt_api tests
ca9025a 
Tests were broken because of the introduction of
the VIP, we were taking the first IP listed matching
the CIDR, but VIP will always be the first as this
is the first IP in the subnet.

Refs: #1687
alexandre-allard added a commit that referenced this issue Nov 21, 2019
@alexandre-allard
ci: add a script to add a new metalk8s node
68424c5 
Script can be used to add a new node in an
already running cluster.
Script is needed in the restoration script tests
as we need at least 3 nodes in ETCD cluster to be
able to restore the bootstrap node.

Refs: #1687
alexandre-allard added a commit that referenced this issue Nov 21, 2019
@alexandre-allard
ci: add bootstrap restoration test
c2afd8b 
Refs: #1687
@alexandre-allard alexandre-allard mentioned this issue Nov 21, 2019
Merged
alexandre-allard added a commit that referenced this issue Nov 21, 2019
@alexandre-allard
ci: add bootstrap restoration test
2a0930b 
Refs: #1687
alexandre-allard added a commit that referenced this issue Nov 21, 2019
@alexandre-allard
ci: add bootstrap restoration test
2bec115 
Refs: #1687
alexandre-allard added a commit that referenced this issue Nov 22, 2019
@alexandre-allard
ci: add bootstrap restoration test
94965b3 
Refs: #1687
alexandre-allard added a commit that referenced this issue Nov 24, 2019
@alexandre-allard
ci: add a script to add a new metalk8s node
e2ab665 
Script can be used to add a new node in an
already running cluster.
Script is needed in the restoration script tests
as we need at least 3 nodes in ETCD cluster to be
able to restore the bootstrap node.

Refs: #1687
alexandre-allard added a commit that referenced this issue Nov 24, 2019
@alexandre-allard
ci: add bootstrap restoration test
04b2bb2 
Refs: #1687
alexandre-allard added a commit that referenced this issue Nov 24, 2019
@alexandre-allard
ci: add bootstrap restoration test
e101352 
Refs: #1687
alexandre-allard added a commit that referenced this issue Nov 25, 2019
@alexandre-allard
ci: add bootstrap restoration test
3cada92 
Refs: #1687
TeddyAndrieux pushed a commit that referenced this issue May 19, 2020
@alexandre-allard @TeddyAndrieux
tests/ci: add tests scenario for bootstrap restoration
6d9618a 
This scenario run the restore script and ensure
that everything is working as expected (pods, ...)
after the restoration.

Refs: #1687
TeddyAndrieux added a commit that referenced this issue May 19, 2020
@TeddyAndrieux
ci: add bootstrap restore tests in CI
40e476c 
Launch bootstrap restore tests in CI during post-merge stages, this CI
step first need a full MetalK8s cluster with at least 3-node etcd
cluster

Refs: #1687
TeddyAndrieux added a commit that referenced this issue May 19, 2020
@TeddyAndrieux
ci: add bootstrap restore tests in CI
75cd6d4 
Launch bootstrap restore tests in CI during post-merge stages, this CI
step first need a full MetalK8s cluster with at least 3-node etcd
cluster

Refs: #1687
TeddyAndrieux added a commit that referenced this issue May 19, 2020
@TeddyAndrieux
ci: add bootstrap restore tests in CI
ecc897f 
Launch bootstrap restore tests in CI during post-merge stages, this CI
step first need a full MetalK8s cluster with at least 3-node etcd
cluster

Refs: #1687
TeddyAndrieux pushed a commit that referenced this issue May 20, 2020
@alexandre-allard @TeddyAndrieux
tests/ci: add a second node to expansion tests if node2
c6a3bee 
Deploy two node instead of only one during
expansion tests to have HA etcd and being able to test the
bootstrap restoration

Refs: #1687
TeddyAndrieux pushed a commit that referenced this issue May 20, 2020
@alexandre-allard @TeddyAndrieux
tests/ci: pass backup archive env var to pytest
5afd6b7 
Refs: #1687
TeddyAndrieux pushed a commit that referenced this issue May 20, 2020
@alexandre-allard @TeddyAndrieux
tests: add function to get node control plane ip
1167e1a 
Add a function to help retrieve control plane IP
of a node through SSH using salt-call command.
This function is needed in order to retrieve
the IP of an API Server for the restoration of
the bootstrap

Refs: #1687
TeddyAndrieux pushed a commit that referenced this issue May 20, 2020
@alexandre-allard @TeddyAndrieux
tests: move check_etcd_role definition to conftest
764864c 
This `then` will be useful in restoration tests also
so we move it to a common place

Refs: #1687
TeddyAndrieux pushed a commit that referenced this issue May 20, 2020
@alexandre-allard @TeddyAndrieux
tests: no longer check kube-dns node
054e330 
we don't want to check presence of kube-dns pods
on the bootstrap node, because pods can move
to other node, especially during restore tests

Refs: #1687
TeddyAndrieux pushed a commit that referenced this issue May 20, 2020
@alexandre-allard @TeddyAndrieux
tests/ci: add tests scenario for bootstrap restoration
54f61eb 
This scenario run the restore script and ensure
that everything is working as expected (pods, ...)
after the restoration.

Refs: #1687
TeddyAndrieux added a commit that referenced this issue May 20, 2020
@TeddyAndrieux
ci: add bootstrap restore tests in CI
09bab57 
Launch bootstrap restore tests in CI during post-merge stages, this CI
step first need a full MetalK8s cluster with at least 3-node etcd
cluster

Refs: #1687
TeddyAndrieux pushed a commit that referenced this issue May 20, 2020
@alexandre-allard @TeddyAndrieux
tests: move check_etcd_role definition to conftest
08b88a4 
This `then` will be useful in restoration tests also
so we move it to a common place

Refs: #1687
TeddyAndrieux pushed a commit that referenced this issue May 20, 2020
@alexandre-allard @TeddyAndrieux
tests: no longer check kube-dns node
49a28de 
we don't want to check presence of kube-dns pods
on the bootstrap node, because pods can move
to other node, especially during restore tests

Refs: #1687
TeddyAndrieux pushed a commit that referenced this issue May 20, 2020
@alexandre-allard @TeddyAndrieux
tests/ci: add tests scenario for bootstrap restoration
27e0f63 
This scenario run the restore script and ensure
that everything is working as expected (pods, ...)
after the restoration.

Refs: #1687
TeddyAndrieux added a commit that referenced this issue May 20, 2020
@TeddyAndrieux
ci: add bootstrap restore tests in CI
bd08570 
Launch bootstrap restore tests in CI during post-merge stages, this CI
step first need a full MetalK8s cluster with at least 3-node etcd
cluster

Refs: #1687
TeddyAndrieux pushed a commit that referenced this issue May 25, 2020
@alexandre-allard @TeddyAndrieux
tests/ci: add a second node to expansion tests if node2
774ce0f 
Deploy two node instead of only one during
expansion tests to have HA etcd and being able to test the
bootstrap restoration

Refs: #1687
TeddyAndrieux pushed a commit that referenced this issue May 25, 2020
@alexandre-allard @TeddyAndrieux
tests/ci: pass backup archive env var to pytest
2177dc4 
Refs: #1687
TeddyAndrieux pushed a commit that referenced this issue May 25, 2020
@alexandre-allard @TeddyAndrieux
tests: move check_etcd_role definition to conftest
e194582 
This `then` will be useful in restoration tests also
so we move it to a common place

Refs: #1687
TeddyAndrieux pushed a commit that referenced this issue May 25, 2020
@alexandre-allard @TeddyAndrieux
tests: no longer check kube-dns node
ca8b522 
we don't want to check presence of kube-dns pods
on the bootstrap node, because pods can move
to other node, especially during restore tests

Refs: #1687
TeddyAndrieux pushed a commit that referenced this issue May 25, 2020
@alexandre-allard @TeddyAndrieux
tests/ci: add tests scenario for bootstrap restoration
4b7a967 
This scenario run the restore script and ensure
that everything is working as expected (pods, ...)
after the restoration.

Refs: #1687
TeddyAndrieux added a commit that referenced this issue May 25, 2020
@TeddyAndrieux
ci: add bootstrap restore tests in CI
0293ddd 
Launch bootstrap restore tests in CI during post-merge stages, this CI
step first need a full MetalK8s cluster with at least 3-node etcd
cluster

Refs: #1687
This was referenced Sep 15, 2020
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alexandre-allard alexandre-allard

@TeddyAndrieux TeddyAndrieux

Labels
complexity:medium Something that requires one or few days to fix topic:tests What's not tested may be broken
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Test bootstrap restore in CI scality/metalk8s
4 participants
@thomasdanan @alexandre-allard @TeddyAndrieux @gdemonet