Fix containerized 3.9 -> 3.10 upgrade #8239

vrutkovs · 2018-05-02T20:47:12Z

Node is now a system container
Rework a few etcd parts to properly backup and migrate to a static pod

Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1571724

TODO:

Fix etcd backup and update to static pods for Atomic
Update ends with

TASK [fail] ***************************************************************************************************************************************************
task path: /code/openshift-ansible/playbooks/common/openshift-cluster/upgrades/pre/config.yml:63
Wednesday 02 May 2018  20:48:53 +0000 (0:00:00.040)       0:23:37.982 ********* 
fatal: [vrutkovs_4d58cb0034c4-master-1]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "msg": "Master running 3.9.0 must be upgraded to 3.10 before node upgrade can be run."
}

openshift-ci-robot · 2018-05-02T20:47:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vrutkovs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [vrutkovs]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

michaelgugino · 2018-05-02T21:26:21Z

roles/etcd/defaults/main.yaml

@@ -89,7 +91,7 @@ etcd_listen_client_urls: "{{ etcd_url_scheme }}://{{ etcd_ip }}:{{ etcd_client_p
 #etcd_peer: 127.0.0.1
 etcdctlv2: "{{ r_etcd_common_etcdctl_command }} --cert-file {{ etcd_peer_cert_file }} --key-file {{ etcd_peer_key_file }} --ca-file {{ etcd_peer_ca_file }} -C https://{{ etcd_peer }}:{{ etcd_client_port }}"

-etcd_service: etcd
+etcd_service: "{{ (r_etcd_common_etcd_runtime == 'docker') | ternary('etcd_container', 'etcd') }}"


I don't think we need this. We're brute-force removing the services in favor of pods, except for stand-alone rpm etcd.

Yep, this currently breaks, so I'll remove etcd commits

michaelgugino · 2018-05-02T21:29:15Z

roles/openshift_node/tasks/systemd_units.yml

@@ -13,5 +14,14 @@
  - reload systemd units
  - restart node

+- block:
+  - name: Remove existing systemd service


I'm not sure if this is what we want?

If the node is already a system container (which was possible previously in all types of deploys), and we remove this file and the host was already updated (IE, this is a no-op because this host was already upgraded) I'm not sure that the service unit will be placed again via the atomic command.

If the node is already a system container (which was possible previously in all types of deploys)

On Atomic installs this was a different node service. As a result atomic install would crash and complain that service file already exists. Atomic install is not very smart though - it won't create a service file if its missing during a second run.

This part is WIP still

Reworked this to remove the services only when node is not yet bootstrapped

openshift-ci-robot · 2018-05-04T13:07:31Z

@vrutkovs: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/openshift-jenkins/system-containers	`4b08c5c`	link	`/test system-containers`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

* Fix etcd runtime detection when setting up etcd * During upgrade etcd runtime should be detected using systemd services status * Mask, disable and stop services before removing service files * System container should be removed as stopping service doesn't seem to cut it * etcd cluster health check would wait for etcd static pod container to start

sdodson

Also, I'd discussed w/ Clayton copying all the variables from /etc/sysconfig/{{ openshift_service_type }}-api to /etc/origin/master/master.env because at least in free-int we lost AWS configuration because that was added manually to /etc/sysconfig/atomic-openshift-master-api.

What do you think?

sdodson · 2018-05-04T14:44:57Z

roles/openshift_control_plane/tasks/static.yml

+    state: absent
+  with_items:
+  - /etc/sysconfig/origin-master-api
+  - /etc/sysconfig/origin-master-controllers


- /etc/sysconfig/{{ openshift_service_type }}-api - /etc/sysconfig/{{ openshift_service_type }}-controllers

sdodson · 2018-05-04T14:50:23Z

BTW, migrating anything from /etc/sysconfig/atomic-openshift-master-* to master.env we can look at in a follow up.

vrutkovs · 2018-05-04T14:56:29Z

we lost AWS configuration because that was added manually to /etc/sysconfig/atomic-openshift-master-api

This seems to be set in current master.env. Lets fix the missing vars later on, as I'm not sure which ones need to be migrated and which are not required anymore (e.g. OPTIONS and OPENSHIFT_DEFAULT_REGISTRY are probably not needed anymore)

sdodson

/lgtm

all concerns seem to have been addressed

vrutkovs · 2018-05-04T16:24:21Z

bot, retest this please

vrutkovs · 2018-05-04T17:09:17Z

bot, retest this please

sdodson · 2018-05-04T17:53:25Z

We need to dig into the flake we're seeing here, discussion on that at #8264

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 2, 2018

openshift-ci-robot requested review from michaelgugino and sdodson May 2, 2018 20:47

openshift-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label May 2, 2018

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 2, 2018

michaelgugino previously requested changes May 2, 2018

View reviewed changes

vrutkovs force-pushed the containerized-3.9-to-3.10 branch 2 times, most recently from 400a23d to 7daae2a Compare May 3, 2018 11:04

vrutkovs mentioned this pull request May 3, 2018

[WIP] Fix etcd runtime detection to fix upgrade 3.9 -> 3.10 #8253

Closed

vrutkovs force-pushed the containerized-3.9-to-3.10 branch from e5ee8b8 to 7010f57 Compare May 3, 2018 14:31

openshift-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels May 3, 2018

vrutkovs force-pushed the containerized-3.9-to-3.10 branch from 4b08c5c to b3dd1f8 Compare May 3, 2018 21:06

vrutkovs mentioned this pull request May 3, 2018

PAPR: run upgrade from 3.9 branch #8250

Closed

vrutkovs force-pushed the containerized-3.9-to-3.10 branch from b3dd1f8 to 713b401 Compare May 3, 2018 21:51

openshift-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 3, 2018

Setup node in system container when updating 3.9 to 3.10

91b70e5

vrutkovs force-pushed the containerized-3.9-to-3.10 branch 6 times, most recently from 54087ea to 9a893cf Compare May 4, 2018 12:14

vrutkovs changed the title ~~WIP Fix containerized 3.9 -> 3.10 upgrade~~ Fix containerized 3.9 -> 3.10 upgrade May 4, 2018

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 4, 2018

vrutkovs force-pushed the containerized-3.9-to-3.10 branch from 9a893cf to c5666cd Compare May 4, 2018 12:17

vrutkovs force-pushed the containerized-3.9-to-3.10 branch from 373a881 to 7019c2c Compare May 4, 2018 13:24

vrutkovs force-pushed the containerized-3.9-to-3.10 branch 2 times, most recently from 9c6c5b3 to bc70748 Compare May 4, 2018 14:26

openshift-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels May 4, 2018

vrutkovs force-pushed the containerized-3.9-to-3.10 branch from bc70748 to f202390 Compare May 4, 2018 14:37

sdodson requested changes May 4, 2018

View reviewed changes

Remove outdated api/controllers settings in - /etc/sysconfig

7dc9e74

vrutkovs force-pushed the containerized-3.9-to-3.10 branch from f202390 to 7dc9e74 Compare May 4, 2018 14:57

sdodson approved these changes May 4, 2018

View reviewed changes

openshift-ci-robot assigned sdodson May 4, 2018

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label May 4, 2018

sdodson merged commit 17155a2 into openshift:master May 4, 2018

vrutkovs deleted the containerized-3.9-to-3.10 branch May 30, 2018 08:01

Fix containerized 3.9 -> 3.10 upgrade #8239

Fix containerized 3.9 -> 3.10 upgrade #8239

vrutkovs commented May 2, 2018 •

edited

openshift-ci-robot commented May 2, 2018

michaelgugino May 2, 2018

vrutkovs May 3, 2018

michaelgugino May 2, 2018

vrutkovs May 3, 2018

vrutkovs May 4, 2018

openshift-ci-robot commented May 4, 2018 •

edited

sdodson left a comment

sdodson May 4, 2018

vrutkovs May 4, 2018

sdodson commented May 4, 2018

vrutkovs commented May 4, 2018

sdodson left a comment

vrutkovs commented May 4, 2018

vrutkovs commented May 4, 2018

sdodson commented May 4, 2018

Fix containerized 3.9 -> 3.10 upgrade #8239

Fix containerized 3.9 -> 3.10 upgrade #8239

Conversation

vrutkovs commented May 2, 2018 • edited

openshift-ci-robot commented May 2, 2018

michaelgugino May 2, 2018

Choose a reason for hiding this comment

vrutkovs May 3, 2018

Choose a reason for hiding this comment

michaelgugino May 2, 2018

Choose a reason for hiding this comment

vrutkovs May 3, 2018

Choose a reason for hiding this comment

vrutkovs May 4, 2018

Choose a reason for hiding this comment

openshift-ci-robot commented May 4, 2018 • edited

sdodson left a comment

Choose a reason for hiding this comment

sdodson May 4, 2018

Choose a reason for hiding this comment

vrutkovs May 4, 2018

Choose a reason for hiding this comment

sdodson commented May 4, 2018

vrutkovs commented May 4, 2018

sdodson left a comment

Choose a reason for hiding this comment

vrutkovs commented May 4, 2018

vrutkovs commented May 4, 2018

sdodson commented May 4, 2018

vrutkovs commented May 2, 2018 •

edited

openshift-ci-robot commented May 4, 2018 •

edited