Set the server_addr for the dashboard and prometheus modules

[travisn]

Signed-off-by: travisn tnielsen@redhat.com

Description of your changes:
The server_addr needs to be set on the mgr so the prometheus and dashboard mgr modules can bind to the pod ip when starting their endpoints. The mgr key is not sufficient to set this setting, so the admin key is used in the config-init container, then the admin key is deleted before the init container completes.

Which issue is resolved by this Pull Request:
Resolves #2335

Checklist:

• Documentation has been updated, if necessary.
• Pending release notes updated with breaking and/or notable changes, if necessary.
• Upgrade from previous release is tested and upgrade user guide is updated, if necessary.
• Code generation (make codegen) has been run to update object specifications, if necessary.
• Comments have been added or updated based on the standards set in CONTRIBUTING.md

[BlaineEXE]

Looks good.

To note the thought I mentioned in the meeting today: I have been having the thought that I would wish the operator could do more of the config work and we could get rid of the need for calling the rook binary for init. I think this PR is good as-is; I am noticing here that it is a fair bit of additional code to add this support into the binary init path, and it would be nice to not have to keep doing this in the future.

[travisn]

@BlaineEXE @galexrt Per our discussion instead of setting the server_addr to the pod ip in the mgr init container, I successfully tested that setting it to 0.0.0.0 is also working. While this does appear to fix both #2397 and #2171, this would only work for ipv4 clusters. If there is an ipv6 cluster, we should leave the binding to the default [::].

Leaving this PR as-is I believe would work with both ipv4 and ipv6 since we would be binding to the correct pod ip either way. I do prefer the approach of setting 0.0.0.0 centrally in the operator, although this means we need a new setting in the operator.yaml to indicate if we're running ipv4 or ipv6. Any other suggestions?

[galexrt]

Why use fmt.Sprintf() here?

[travisn]

i'll clean this up

[galexrt]

Same here

[galexrt]

allMgrs is empty. It is used in the MgrSetConfig func to build the mgr ID. Is that right?

[travisn]

If it applies to all mgrs, the empty string is needed. I'll rework this to make it more readable.

[galexrt]

We have two other env vars that have the Pod IP in them (see line 99-100) do we really need a new env var here?

[travisn]

while it's not strictly necessary, i wanted to be clear about the purpose of this setting. If the public and private ip configuration changes in the future, we should reevaluate this setting as well. This way it will be more obvious to evaluate it.

[BlaineEXE]

I think I would lean toward the solution that supports both IPv4 and IPv6 without special support for either. I think there could be many ways to do that, though. I'm not sure what might happen if the k8s network is IPv4 and the host network is on IPv6 (or vice versa).

If we know that the mgr binding will always be on the cluster network, we could go the route of setting a config parameter with the 4/6 type of IP for the overlay network, but it would seem better to me to detect that automatically and not make the user configure it at all. Or if we know for certain that the mgr will use the host network, we could detect that type.

A lot of my concerns are based around the fact that many Ceph clusters will likely have 3 networks (mgmt, client, and cluster data). And to be fair, I haven't seen anywhere that Rook has plans for how to achieve that kind of support. And my level of ignorance about this subject is high.

I think my bottom line opinion at present is that if Rook is supposed to support IPv4 and IPv6 now, we keep the initial patch. But if Rook currently just supports IPv4, setting 0.0.0.0 is easier, and we can cross the IPv6 hurdle later when we are ready to tackle it.

[dyusupov]

I second this sentiment. Same applies to EdgeFS - even more so, as it trying to optimize backend networking as much as possible, leveraging UDP/IP and PF-RING. Backend network needs to be isolated from client and mgmt for many not even performance-related reasons too. I was thinking that we should start working on multi-network interface CRD design, perhaps incorporating: https://github.com/intel/multus-cni

…

On Thu, Dec 20, 2018 at 2:10 PM Blaine Gardner ***@***.***> wrote: I think I would lean toward the solution that supports both IPv4 and IPv6 without special support for either. I think there could be many ways to do that, though. I'm not sure what might happen if the k8s network is IPv4 and the host network is on IPv6 (or vice versa). If we know that the mgr binding will always be on the cluster network, we could go the route of setting a config parameter with the 4/6 type of IP for the overlay network, but it would seem better to me to detect that automatically and not make the user configure it at all. Or if we know for certain that the mgr will use the host network, we could detect that type. A lot of my concerns are based around the fact that many Ceph clusters will likely have 3 networks (mgmt, client, and cluster data). And to be fair, I haven't seen anywhere that Rook has plans for how to achieve that kind of support. And my level of ignorance about this subject is high. I think my bottom line at present is that if Rook is supposed to support IPv4 and IPv6, we keep the initial patch. But if Rook currently just supports IPv4, setting 0.0.0.0 is easier, and we can cross the IPv6 hurdle later when we are ready to tackle it. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2408 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABNTVz5FZmW9VYAPjOfBkiy7yVnHPlZYks5u7ArGgaJpZM4ZckT_> .

[BlaineEXE]

I think that bears more looking into, and this seems to meet a lot of my current thoughts for what is needed for host networking. It also seems similar in scope to CNI Genie, which I have been strongly advised to avoid, so I do have some hesitation.

With Ceph at least, we will also be fighting any sort of layering between the pods and the network. @sebastian-philipp has quoted a figure that I now forget that Ceph performance degrades substantially even just by enabling iptables with non-blocking rules on bare metal hosts. I believe it was around 60% performance degradation.

I think for all Rook storage backends, host networking's main feature will be to increase network throughput and reduce latency, so we may want to make some decisions around what Rook wants to support. Do we risk degrading performance by going through CNI layers that are heavy and increasing security, or do we leave the networks as unimpeded as possible and instead give strong recommendations about isolating the different networks for security purposes? And do we want to give the user the option of either?

[dyusupov]

hostNetwork: true - clearly will boost performance when separate backend interface configured. We, however, need to try to avoid the use of hostNetwork as it breaks cross pod isolation. I'm surprised it is just 60% drop in performance... Isolation on the same physical interface will be a challenge and likely will affect performance. True, when CNI is on the data path. With multi-interface pod networking where Ceph or EdgeFS backend networking is running on the dedicated interface, performance will actually improve! Genie is another interesting alternative. But given that Intel is backing Multus - that would be my first choice.

…

On Thu, Dec 20, 2018 at 2:25 PM Blaine Gardner ***@***.***> wrote: I think that bears more looking into, and this seems to meet a lot of my current thoughts for what is needed for host networking. It also seems similar in scope to CNI Genie, which I have been strongly advised to avoid, so I do have some hesitation. With Ceph at least, we will also be fighting any sort of layering between the pods and the network. @sebastian-philipp <https://github.com/sebastian-philipp> has quoted a figure that I now forget that Ceph performance degrades substantially even just by enabling iptables with non-blocking rules on bare metal hosts. I believe it was around 60% performance degradation. I think for all Rook storage backends, host networking's main feature will be to increase network throughput and reduce latency, so we may want to make some decisions around what Rook wants to support. Do we risk degrading performance by going through CNI layers that degrade network throughput/latency and increasing security, or do we leave the network as clean as possible and instead give strong recommendations about isolating the different networks for security purposes? And do we want to give the user the option of either? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2408 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABNTV6_6-uAaa1ZPdjo_-9-_7TDdiy86ks5u7A5PgaJpZM4ZckT_> .

[travisn]

Great discussion around networking. I would propose that this PR go ahead and merge since it is compatible with either ipv4 or ipv6.

[dyusupov]

+1

…

On Thu, Dec 20, 2018 at 3:02 PM Travis Nielsen ***@***.***> wrote: Great discussion around networking. I would propose that this PR go ahead and merge since it is compatible with either ipv4 or ipv6. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2408 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABNTV514IDhbnefoav6pfraRdFv_5v7Cks5u7Bb5gaJpZM4ZckT_> .