sdn: upgrade SDN after all nodes are upgraded.

[squeed]

Restarting OVS kills all pods on the node. So, while we can't do it as part of the node drain-upgrade loop, we can at least do it when users are expecting disruption.

[rjhowe]

During the control-plain upgrade the master node processes are updated, can we guarantee that waiting on a user to upgrade the nodes that the master nodes which were upgraded will still have a working SDN network in the mean time.
I am not sure how we would go about this with the current setup, making use of a daemon set, likely this is the only way without making to many changes.

https://github.com/openshift/openshift-ansible/blob/release-3.11/playbooks/common/openshift-cluster/upgrades/v3_11/upgrade_control_plane_part2.yml#L92-L110

[sdodson]

/lgtm
/cherrypick release-3.10

[openshift-cherrypick-robot]

@sdodson: once the present PR merges, I will cherry-pick it on top of release-3.10 in a new PR and assign it to you.

In response to this:

/lgtm
/cherrypick release-3.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sdodson, squeed

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [sdodson]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest

Please review the full test history for this PR and help us cut down flakes.

[openshift-cherrypick-robot]

@sdodson: #10910 failed to apply on top of branch "release-3.10":

error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	playbooks/common/openshift-cluster/upgrades/upgrade_nodes.yml
A	playbooks/common/openshift-cluster/upgrades/v3_11/upgrade_control_plane_part2.yml
M	roles/openshift_sdn/tasks/main.yml
Falling back to patching base and 3-way merge...
Auto-merging roles/openshift_sdn/tasks/main.yml
Auto-merging playbooks/common/openshift-cluster/upgrades/v3_10/upgrade_control_plane.yml
CONFLICT (content): Merge conflict in playbooks/common/openshift-cluster/upgrades/v3_10/upgrade_control_plane.yml
Auto-merging playbooks/common/openshift-cluster/upgrades/upgrade_nodes.yml
Patch failed at 0001 sdn: upgrade SDN after all nodes are upgraded.

In response to this:

/lgtm
/cherrypick release-3.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.