Configuring service-dns fails when installed from helm chart

[seenimurugan]

When run the Hazelcast server using helm chart with service-dns value configured fails with the following error

Caused by: com.hazelcast.config.InvalidConfigurationException: Properties 'service-dns' and ('service-name' or 'service-label-name') cannot be defined at the same time

I override the config properties from the parent chart as follows

hazelcast:
  cluster:
    memberCount: 2
  service:
    clusterIP: "None"
  hazelcast:
    rest: true
    yaml:
      hazelcast:
        network:
          join:
            multicast:
              enabled: false
            kubernetes:
              enabled: true
              service-dns: ${serviceName}
        management-center:
          enabled: ${hazelcast.mancenter.enabled}
          url: ${hazelcast.mancenter.url}

I tried overriding the service-name property to null but it is still not working.

[mesutcelik]

I tried the same and it does not work. However, when I fetched the helm chart and modified locally, helm install . creates correct ConfigMap.

[mesutcelik]

I checked that one again and I see it is similar to that issue.
helm/helm#5534

Apparently, When you do helm install hazelcast/hazelcast -f values.yaml, helm merges both local and remote yaml file, we need to set service-name: null as @seenimurugan did.

local values.yaml

  yaml:
    hazelcast:
      network:
        join:
          multicast:
            enabled: false
          kubernetes:
            enabled: true
            service-dns: ${serviceName}
            service-name: null
      management-center:
        enabled: ${hazelcast.mancenter.enabled}
        url: ${hazelcast.mancenter.url}

It is good so far but then we receive an exception with that configuration and it is thrown by Hazelcast.

xception in thread "main" com.hazelcast.config.InvalidConfigurationException: The configuration entry under hazelcast/network/join/kubernetes/service-name is null. Please check if the provided YAML configuration is well-indented and no blocks started without sub-nodes.
	at com.hazelcast.config.yaml.YamlDomChecker.reportNullEntryOnConcretePath(YamlDomChecker.java:71)
	at com.hazelcast.config.yaml.YamlDomChecker.check(YamlDomChecker.java:47)
	at com.hazelcast.config.yaml.YamlDomChecker.check(YamlDomChecker.java:50)
	at com.hazelcast.config.yaml.YamlDomChecker.check(YamlDomChecker.java:50)
	at com.hazelcast.config.yaml.YamlDomChecker.check(YamlDomChecker.java:50)
	at com.hazelcast.config.YamlConfigBuilder.parseAndBuildConfig(YamlConfigBuilder.java:153)
	at com.hazelcast.config.YamlConfigBuilder.build(YamlConfigBuilder.java:131)
	at com.hazelcast.config.YamlConfigBuilder.build(YamlConfigBuilder.java:122)
	at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:136)
	at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:91)
	at com.hazelcast.core.server.StartServer.main(StartServer.java:46)

Apparently, we do not allow null passing in yaml files to Hazelcast Configuration.
Do you have any suggestions here @leszko @blazember ?

[mesutcelik]

I had a quick chat with @blazember
I think we need to fix this in hazelcast-kubernetes plugin by introducing a new parameter discovery-mode: API | DNS and deprecating service-dns. If discovery-mode is DNS then service-name can be used instead of service-dns

[leszko]

The workaround for this (or maybe even a proper solution) is to set service-name: to an empty string.

I think we'll not introduce discovery-mode API | DNS, but rather extract the DNS Lookup discovery as a separate modele (or include in hazelcast/hazelcast).

[maurizio-lattuada]

Hi,
I have exactly the same problem of @mesutcelik (#58 (comment)).
No matter I set either

• service-name: null
• service-name: (note the white space after the ":")
• service-name: (no white space after the ":")
• service-name: ~

I get always the exception

InvalidConfigurationException: The configuration entry under hazelcast/network/join/kubernetes/service-name is null. Please check if the provided YAML configuration is well-indented and no blocks started without sub-nodes.

For completeness, I'm using Hazelcast 3.12.6 (same behavior also with the 3.12.5) deployed with its helm chart version 2.10.0 on a Rancher cluster (version 2.2.8).
This is the YAML file passed with custom values:

# Hazelcast custom Helm chart template
---
image:
  repository: "nexus.internal/hazelcast/hazelcast"
  tag: "3.12.6"
cluster:
  memberCount: 1
metrics:
  enabled: true
rbac:
  enabled: false
serviceAccount:
  create: false
mancenter:
  enabled: false
hazelcast:
  yaml:
    hazelcast:
      group:
        name: tomcat
      network:
        join:
          kubernetes:
            enabled: true
            service-dns: ${serviceName}.${namespace}.svc.cluster.local
            service-name: 

(note again the white space after the ":" of service-name).

This is the configmap containing the "hazelcast.yaml" file created by the helm chart:

apiVersion: v1
data:
  hazelcast.yaml: |-
    hazelcast:
      group:
        name: tomcat
      management-center:
        enabled: ${hazelcast.mancenter.enabled}
        url: ${hazelcast.mancenter.url}
      network:
        join:
          kubernetes:
            enabled: true
            namespace: ${namespace}
            resolve-not-ready-addresses: true
            service-dns: ${serviceName}.${namespace}.svc.cluster.local
            service-name: null
          multicast:
            enabled: false
        rest-api:
          enabled: true
          endpoint-groups:
            HEALTH_CHECK:
              enabled: true
kind: ConfigMap
metadata:
  creationTimestamp: "2020-02-12T16:04:44Z"
  labels:
    app.kubernetes.io/instance: hazelcast
    app.kubernetes.io/managed-by: Tiller
    app.kubernetes.io/name: hazelcast
    helm.sh/chart: hazelcast-2.10.0
    io.cattle.field/appId: hazelcast
  name: hazelcast-configuration
  namespace: hazelcast
  resourceVersion: "22916230"
  selfLink: /api/v1/namespaces/hazelcast/configmaps/hazelcast-configuration
  uid: 62b1514b-4db1-11ea-b80a-00505682af12

You can see that service-name has been automagically assigned the null value.

Of course, by setting either:

• service-name: ""
• service-name: '' (two single quotes)
• leaving completely away the service-name entry

Hazelcast claims about the incorrect usage of service-name together with service-dns. This means that the yaml passed above is syntactically correct.

Any idea/feedback?

[leszko]

I think we need to fix it in the hazelcast-kubernetes plugin to treat empty strings the same as null. Added a "bug" label.

[leszko]

Created an issue in hazelcast-kubernetes: hazelcast/hazelcast-kubernetes#197

[leszko]

The workaround for this issue is to create ConfigMap with Hazelcast configuration manually and use existingConfigMap.

[mesutcelik]

@leszko I see k8s issue is merged and closed so can we close this issue? Do we need any readme update?

[leszko]

Let's close it when Hazelcast 4.1 is released and Helm Chart is updated.

[leszko]

Fixed by #170

It should work fine starting from the chart version 3.5.0