Get exception "Unknown protocol: HTT" when using hazelcast-hibernate with hazelcast-kubernetes

[cyj5230]

I used https://github.com/Cepr0/sb-hazelcast-hibernate-l2c-demo as reference, which works well, but when I added hazelcast-kubenetes by this guide, I get following exception and cannot run the pod.

2020-01-10 02:41:55.147  WARN 1 --- [.IO.thread-in-1] com.hazelcast.nio.tcp.TcpIpConnection    : [172.20.1.2]:5701 [hl-event-service] [3.12.4] Connection[id=9, /172.20.1.2:41156->/172.20.1.236:80, qualifier=null, endpoint=[172.20.1.236]:80, alive=false, type=NONE] closed. Reason: Exception in Connection[id=9, /172.20.1.2:41156->/172.20.1.236:80, qualifier=null, endpoint=[172.20.1.236]:80, alive=true, type=NONE], thread=hz.hl-event-service-instance.IO.thread-in-1

java.lang.IllegalStateException: Unknown protocol: HTT
	at com.hazelcast.nio.tcp.UnifiedProtocolDecoder.onRead(UnifiedProtocolDecoder.java:107)
	at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:135)
	at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:369)
	at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:354)
	at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:280)
	at com.hazelcast.internal.networking.nio.NioThread.run(NioThread.java:235)

Here are my configs (slightly different from the reference)
HazelcastConfiguration.java

@Bean
  public Config config() {
    Config hazelcastConfig = new Config();
    hazelcastConfig.setInstanceName(properties.getInstance());
    hazelcastConfig.getGroupConfig().setName(properties.getGroup());

    JoinConfig joinConfig = hazelcastConfig.getNetworkConfig().getJoin();
    joinConfig.getMulticastConfig().setEnabled(false);
    joinConfig.getKubernetesConfig().setEnabled(true);
    return hazelcastConfig;
  }

application.yml

hibernate:
        ddl-auto: none
        show-sql: true
        hbm2ddl:
          auto: validate
        temp:
          use_jdbc_metadata_defaults: false
        cache:
          use_query_cache: true
          use_second_level_cache: true
          hazelcast:
            instance_name: ${hazelcast-custom.instance}
          region:
            factory_class: com.hazelcast.hibernate.HazelcastLocalCacheRegionFactory

hazelcast-custom:
  instance: hl-event-service-instance
  group: hl-event-service

pom.xml (use Hazelcast 3 because hazelcast-hibernate53 doesn't fit Hazelcast 4 yet)

<hazelcast.version>3.12.4</hazelcast.version>
<hazelcast-hibernate53.version>1.3.2</hazelcast-hibernate53.version>
<hazelcast-kubernetes.version>1.5.2</hazelcast-kubernetes.version>

Where does the "HTT" come from? Is there any crash in the config for hazelcast-hibernate and hazelcast-kubernetes?
I noticed an issue in hibernate with similar exception.

[cyj5230]

Solved.
Should set service-name if there are other services in the same namespace.

[letanloc1998]

Hi, could you explain how to set service-name. Thank you very much

[leszko]

You can set it in the Hazelcast configuration in the Kubernetes section. Check this: https://github.com/hazelcast/hazelcast-kubernetes#hazelcast-configuration

[emillundstrm]

I also had this error, but for a different reason. I had forgotten to add

- containerPort: 5701

in my pod deployment. This apparently caused hazelcast-kubernetes to connect to the only other open port on the pod, which in my case was 8080, where a web application was being served.

Perhaps this will help someone along the line.

[donatelloOo]

What is the default port used when not specifying any service ?
The documentation says:

service-name: service name used to scan only PODs connected to the given service;
if not specified, then all PODs in the namespace are checked

Does it search for 5701 (hardcoded), or a port named "hazelcast" ?
Is there any configuration related to this?

[leszko]

@donatelloOo You can specify the parameter service-port. If not specified, the plugin will try to find the containerPort. If nothing found, then it defaults to 5701. You can check the exact logic here and here.

[imjuoy]

Also check if you have the service.yaml properly configured, if you are on kubernetes.

[nightswimmings]

@imjuoy This is in case you are discovering through services (service-name/label), but if you rely on default fallback method which retrieves pod ips directly then it is not a matter of deployment nor service yaml configuration. In that case the plugin still tries to find containers spec ports on the yaml and if you don't specify service-port property then you get 8080 or whatever main server port you defined as target in those yaml. Remember that Kubernetes allows connecting to any port on the container if you do it through internal network (just like docker expose), so there can be setups where ports defined in yamls are irrelevant