kops 1.16.0 fails to provision k8s 1.16.8 masters with Amazon Linux 2

[cjbehm]

1. What kops version are you running? The command kops version, will display
this information.

Version 1.16.0

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

Testing upgrade from 1.15.10 to 1.16.8

3. What cloud provider are you using?

AWS (using latest Amazon Linux 2 AMI)

4. What commands did you run? What is the simplest way to reproduce this issue?

kops edit cluster --name <clustername> --state s3://<bucket>
(set version)
kops update cluster --name <clustername> --state s3://<bucket>
kops update cluster --name <clustername> --state s3://<bucket> --yes
kops rolling-update cluster --name <clustername> --state s3://<bucket> --instance-group master-us-east-1f-1 --yes

5. What happened after the commands executed?

This eventually times out because the master node fails a selinux dependency.

From /var/log/messages on the new master

nodeup: selinux-policy >= 3.13.1-216.el7 is needed by container-selinux-2:2.107-1.el7_6.noarch
nodeup: selinux-policy-base >= 3.13.1-216.el7 is needed by container-selinux-2:2.107-1.el7_6.noarch
nodeup: selinux-policy-targeted >= 3.13.1-216.el7 is needed by container-selinux-2:2.107-1.el7_6.noarch

6. What did you expect to happen?

The node to provision and become healthy.

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

I am able to use the same kops version to revert back to 1.15.10, following the same set of commands.

This seems like it might be related to these two commits, based on a search for container-selinux in the repo

2a6aeaf#diff-a0fc0755ee3c8b91f928e7c0c517d906

5f93068#diff-a0fc0755ee3c8b91f928e7c0c517d906

I ran this both from the kops downloaded with homebrew as well as kops downloaded directly from the kops releases in case there was some discrepancy, but the result was the same.

[hakman]

This was fixed in Kops 1.18. You should be able to try it in 1.18 beta.
/close

[k8s-ci-robot]

@hakman: Closing this issue.

In response to this:

This was fixed in Kops 1.18. You should be able to try it in 1.18 beta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[hakman]

@cjbehm Could you try setting the Docker version to 18.06.3 and try to see if it works?
Kops 1.18 has support for installing Docker from .tgz package instead of .rpm to avoid the selinux policy. That part cannot be backported.

[cjbehm]

@hakman thank you for the suggestion, that allowed the provisioning to go through. I definitely appreciate the follow up with a potential workaround!

For anyone who comes across this with the same problem

spec:
  docker:
    version: 18.06.3