baremetal: ipv6, switch to NM dispatcher for DNS VIP prepending

[hardys]

The prepend via dhclient doesn't work via ipv6, so switch to a
NetworkManager dispatcher that runs after dhclient instead as a
workaround.

- What I did

Reworked the prepender implementation (with help from @celebdor) to avoid using the prepend domain-name-servers option in dhclient.conf - it seems this option only works for ipv4[1] and the suggested alternative of prepend dhcp6.name-servers also doesn't seem to work when the resolv.conf is managed by NetworkManager.

Instead we configure NetworkManager to no longer manage the resolv.conf directly, and rely on a dispatcher script which injects the necessary IP on master/worker nodes to correctly reference coredns for the baremetal platform.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=643890

- How to verify it

Deploy with ipv4 and ipv6, confirm that on the masters the DNS VIP is prepended to the resolv.conf, and on the workers the local nic IP for the controlplane network (not the DNS VIP) is configured.

Also check sudo journalctl -b | grep prepender to see the log output from the dispatcher scripts.

- Description for the changelog

For the baremetal platform management of the resolv.conf is now handled via a NetworkManager dispatcher script, so that the necessary DNS server can be prepended for both ipv4 and ipv6 environments.

[celebdor]

/lgtm

[celebdor]

/assign yboaron

[celebdor]

/assign @rgolangh
/assign @Fedosin

You may want to copy the approach (specially looking to IPv6 support).

[rgolangh]

This looks good and will address the prepend issues that I saw #1313 - I'm going to add this to oVirt.

[runcom]

/retest

[hardys]

/retest

[openshift-ci-robot]

@hardys: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-scaleup-rhel7	33df2e0	link	/test e2e-aws-scaleup-rhel7

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[runcom]

/skip

[runcom]

/retest

[kikisdeliveryservice]

This is passing tests can we get a review @bcrochet / @celebdor ?

[hardys]

This is passing tests can we get a review @bcrochet / @celebdor ?

Also cc @russellb @derekhiggins who have tested this and found it to work OK - I've also re-tested it in a regular ipv4 environment.

[bcrochet]

/lgtm

[celebdor]

/lgtm

[hardys]

@celebdor @bcrochet @russellb I think we may have to make this also apply to the bootstrap VM, particularly because the NM conf where we set rc-manager=unmanaged in this PR is common?

Also we set the prepend dhclient.conf option in the installer here:

https://github.com/openshift/installer/blob/master/data/data/bootstrap/baremetal/files/etc/dhcp/dhclient.conf#L7

So I think we need to remove that and ensure the prepender gets added for all nodes including bootstrap?

There are only common, master and worker templates directories, how does the precedence work, e.g can we have a common template (which will be for the bootstrap, or perhaps bootstrap and masters if we switch to using the DNS VIP on the bootstrap VM), then override it in the worker directory with a file of the same name?

[russellb]

@celebdor @bcrochet @russellb I think we may have to make this also apply to the bootstrap VM, particularly because the NM conf where we set rc-manager=unmanaged in this PR is common?

Also we set the prepend dhclient.conf option in the installer here:

https://github.com/openshift/installer/blob/master/data/data/bootstrap/baremetal/files/etc/dhcp/dhclient.conf#L7

So I think we need to remove that and ensure the prepender gets added for all nodes including bootstrap?

There are only common, master and worker templates directories, how does the precedence work, e.g can we have a common template (which will be for the bootstrap, or perhaps bootstrap and masters if we switch to using the DNS VIP on the bootstrap VM), then override it in the worker directory with a file of the same name?

It looks like the MCO templates don't apply to the bootstrap VM. I think we need to add what's needed to the installer.

[kikisdeliveryservice]

@hardys @russellb
Just double check I can approve this for merge though right??

[russellb]

@hardys @russellb
Just double check I can approve this for merge though right??

Yes. There's a related change still needed, but for the bootstrap VM, and that'll be in the installer. This PR is still ready to go.

[kikisdeliveryservice]

Thanks for the update @russellb !

[kikisdeliveryservice]

/cherrypick release-4.3

[openshift-cherrypick-robot]

@kikisdeliveryservice: once the present PR merges, I will cherry-pick it on top of release-4.3 in a new PR and assign it to you.

In response to this:

/cherrypick release-4.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bcrochet, celebdor, hardys, kikisdeliveryservice

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [kikisdeliveryservice]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-cherrypick-robot]

@kikisdeliveryservice: new pull request created: #1406

In response to this:

/cherrypick release-4.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.