Skip to content
This repository has been archived by the owner on Sep 30, 2020. It is now read-only.

Revert etcd upgrade #1555

Merged

Conversation

omar-nahhas
Copy link
Contributor

@omar-nahhas omar-nahhas commented Feb 12, 2019

Etcd was upgraded from version 3.2.13 to version 3.3.10 on this PR e8b7380.

After testing an upgrade from a cluster created with kube-aws 0.12.X to kube-aws master, we have found that, if there is a problem during the upgrade, cloud formation is not able to rollback cleanly, and etcd cluster is left on an unavailable state due to being unable to downgrade etcd.

This is PR Is to revert the etcd upgrade version back to 3.2.13 until a two ways migration is implemented.

kube_aws_etcd_rollback_failed

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Feb 12, 2019
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: c-knowles

If they are not already assigned, you can assign the PR to them by writing /assign @c-knowles in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Feb 12, 2019
@omar-nahhas
Copy link
Contributor Author

/assign @davidmccormick

@davidmccormick
Copy link
Contributor

@omar-nahhas thanks for your testing efforts and work identifying this issue! I agree, let's keep etcd on a stable version until we can implement a smoother path forward with upgrades.

@codecov-io
Copy link

Codecov Report

Merging #1555 into master will decrease coverage by 0.01%.
The diff coverage is 20%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1555      +/-   ##
==========================================
- Coverage   25.47%   25.46%   -0.02%     
==========================================
  Files          97       97              
  Lines        5016     5019       +3     
==========================================
  Hits         1278     1278              
- Misses       3591     3594       +3     
  Partials      147      147
Impacted Files Coverage Δ
pkg/api/plugin.go 0% <ø> (ø) ⬆️
pkg/model/stack_new.go 32.29% <0%> (-0.62%) ⬇️
pkg/api/etcd.go 34.02% <100%> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6b64cf6...6d3d78f. Read the comment docs.

@davidmccormick davidmccormick added this to the v0.13.0 milestone Feb 12, 2019
@davidmccormick davidmccormick merged commit 13f34bf into kubernetes-retired:master Feb 12, 2019
@davidmccormick
Copy link
Contributor

Many thanks for your investigation and fix! 🙏

@omar-nahhas
Copy link
Contributor Author

Here are some results of manual testing I have done:
1 - create the cluster using kube 0.12
test_step_1_version_12

2 - try upgrade to kube 0.13, cause a rollback, and observe a successful rollback.
test_step_3_rollback_success

3 - cluster is healthy after rollback
cluster_healthy

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants