Nodes cannot join into cluster automatically using emqx-operator in k8s

[xgxshtc]

What happened?

Nodes cannot join into cluster using emqx-operator in k8s. 3 nodes are started successfully but not in one cluster.
NAME READY STATUS RESTARTS AGE
emqx-core-6d94b88df5-0 1/1 Running 0 14m
emqx-core-6d94b88df5-1 1/1 Running 0 14m
emqx-core-6d94b88df5-2 1/1 Running 0 14m

emqx@emqx-core-6d94b88df5-1:/opt/emqx$ emqx_ctl cluster status
Cluster status: #{running_nodes =>
['emqx@emqx-core-6d94b88df5-1.emqx-headless.dev.svc.cluster.local'],
stopped_nodes => []}

What did you expect to happen?

3 nodes should join the cluster.

How can we reproduce it (as minimally and precisely as possible)?

No response

Anything else we need to know?

emqx operator version: 2.2.6
emqx.yaml 
apiVersion: apps.emqx.io/v2beta1
kind: EMQX
metadata:
  labels:
    app: emqx
  name: emqx
  namespace: dev
spec:
  image: "xxxxxxxxx/emqx/emqx:5.5.0"
  imagePullSecrets:
    - name: xxxxxx
  config:
    data: |
      log {
        file_handlers.default {
          level = info
          file = "log/emqx.log"
        }
      }
  coreTemplate:
    spec:
      replicas: 3
      resources:
        requests:
          cpu: 250m
          memory: 512Mi
  dashboardServiceTemplate:
    metadata:
      annotations:
        service.beta.kubernetes.io/alicloud-loadbalancer-address-type: intranet
        service.beta.kubernetes.io/alicloud-loadbalancer-force-override-listeners: 'true'
        service.beta.kubernetes.io/alicloud-loadbalancer-id: lb-xxxxxxxxxx
    spec:
      type: LoadBalancer
      selector:
        app: emqx
  listenersServiceTemplate:
    metadata:
      annotations:
        service.beta.kubernetes.io/alicloud-loadbalancer-address-type: intranet
        service.beta.kubernetes.io/alicloud-loadbalancer-force-override-listeners: 'true'
        service.beta.kubernetes.io/alicloud-loadbalancer-id: lb-xxxxxxxxx
    spec:
      type: LoadBalancer
      selector:
        app: emqx

EMQX version

$ ./bin/emqx_ctl broker
sysdescr  : EMQX
version   : 5.5.0
datetime  : 2024-03-04T06:25:25.783734115+00:00
uptime    : 38 seconds

OS version

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ uname -a
Linux emqx-core-6d94b88df5-1 5.10.134-15.1.2.lifsea8.x86_64 #1 SMP Tue Aug 29 07:26:14 UTC 2023 x86_64 GNU/Linux

Log files

EMQX_API_KEY__BOOTSTRAP_FILE [api_key.bootstrap_file]: /opt/emqx/data/bootstrap_api_key EMQX_DASHBOARD__LISTENERS__HTTP__BIND [dashboard.listeners.http.bind]: 18083 EMQX_RPC__PORT_DISCOVERY [rpc.port_discovery]: manual EMQX_CLUSTER__DNS__RECORD_TYPE [cluster.dns.record_type]: srv EMQX_CLUSTER__DNS__NAME [cluster.dns.name]: emqx-headless.dev.svc.cluster.local EMQX_CLUSTER__DISCOVERY_STRATEGY [cluster.discovery_strategy]: dns EMQX_NODE__ROLE [node.role]: core EMQX_NODE__DATA_DIR [node.data_dir]: data EMQX_NODE__COOKIE [node.cookie]: ****** EMQX_NODE__NAME [node.name]: emqx@emqx-core-6d94b88df5-0.emqx-headless.dev.svc.cluster.local Listener ssl:default on :8883 started. Listener tcp:default on :1883 started. Listener ws:default on :8083 started. Listener wss:default on :8084 started. Listener http:dashboard on :18083 started. EMQX 5.5.0 is running now!

[id]

ping @Rory-Z

[Rory-Z]

Hi @xgxshtc Could you please try the latest emqx operator?
And looks you deploy emqx in Alibaba cloud, could you check what kinds of Kubernetes service? seems the Alibaba cloud has three kinds of Kubernetes service, and they have some different.