Allow CoherenceCluster crd to override dnsPolicy and enable hostNetwork in statefulsets

[jtyl2017]

Corporate policy may restrict the creation of Kubernetes Services that are of type: LoadBalancer. and as a result, external traffic to kubernetes coherence cluster nodes will be blocked. To allow external traffic access (in cases such as management and extend role nodes), we will need to apply the following workaround.

We will need the ability to set:

• dnsPolicy=ClusterFirstWithHostNet
• hostNetwork=true
• per role annotations
  via the CoherenceCluster crd yaml file.

e.g. Sample content in CoherenceCluster crd yaml file (see ### <<---)

apiVersion: v1
items:
- apiVersion: coherence.oracle.com/v1
  kind: CoherenceCluster
  metadata:
    ...
  spec:
    application:
      ...
    coherence:
      ...
    env:
    ...
    roles:
    - application:
      role: proxy
      network:                              ### <<--- network section
        annotations:                        ### <<--- role specific annotations section
          dns.alpha.kubernetes.io/internal: ### <<--- role specific management coherence url here
        dnsPolicy: ClusterFirstWithHostNet  ### <<--- Chosen DNS Policy here
        hostNetwork:                        ### <<--- Enable HostNetwork
          enabled: true                     ### <<--- Enable HostNetwork

Once the CoherenceCluster crd yaml with above additions are deployed, the:

• selected dnsPolicy,
• hostNetwork enabled
• dns annotations (role specific)
  settings should be applied to resulting Kubernetes Statefulsets and pods created by the CoherenceCluster crd.

e.g. Sample Result from CoherenceCluster crd yaml spec file (see ### <<---)

apiVersion: apps/v1
kind: StatefulSet
metadata:
  ...
  labels:
    ...
  name: ...
spec:
  ...
  template:
    metadata:
      annotations:                        ### <<-- role specific annotations section
        dns.alpha.kubernetes.io/internal: ### <<--- role specific management coherence url here
        prometheus.io/port: "2408"
    spec:
      ...
      containers:
        ...
      dnsPolicy: ClusterFirstWithHostNet  ### <<--- DNS Policy chosen
      hostNetwork: true                   ### <<--- hostNetwork enabled

[thegridman]

The CoherenceCluster CRD already allows for annotations to be added to both StatefulSet/Pod and to the Services created for ports.
Pod/StatefulSet Annotation docs
Service Annotation Docs

For example:

apiVersion: coherence.oracle.com/v1
kind: CoherenceCluster
metadata:
  name: test-cluster
spec:
  annotations:    # <--- annotation here will be added to the StatefuSet/Pod for ALL roles
  roles:
    - role: data
      annotations:    # <--- annotation here will be added to the StatefuSet/Pod for just the data role
        dns.alpha.kubernetes.io/internal:
    - role: proxy
      annotations:    # <--- annotation here will be added to the StatefuSet/Pod for just the proxy role
        dns.alpha.kubernetes.io/internal:

The other settings are not currently supported but they should be simple enough to add.

[thegridman]

Fixed in version 2.0.1

Additional network and DNS configuration has been added to the CRD.
See the docs: https://oracle.github.io/coherence-operator/docs/2.0.1/#/clusters/095_networking