emqx 5.3.2 with emqx-operator won't start on k8s

[jiangwensm]

What happened?

new replicant Pod won't start after editing File Acl, looks like it won't copy acl file data/authz/acl.conf from other Pods

2024-01-04T06:01:35.128157+00:00 [alert] explain: No such file or directory, msg: failed_to_read_acl_file, path: data/authz/acl.conf
EMQX_API_KEY__BOOTSTRAP_FILE [api_key.bootstrap_file]: /opt/emqx/data/bootstrap_api_key
EMQX_DASHBOARD__LISTENERS__HTTP__BIND [dashboard.listeners.http.bind]: 18083
EMQX_RPC__PORT_DISCOVERY [rpc.port_discovery]: manual
EMQX_CLUSTER__DNS__RECORD_TYPE [cluster.dns.record_type]: srv
EMQX_CLUSTER__DNS__NAME [cluster.dns.name]: emqx5-test-headless.tool.svc.cluster.local
EMQX_CLUSTER__DISCOVERY_STRATEGY [cluster.discovery_strategy]: dns
EMQX_NODE__ROLE [node.role]: replicant
EMQX_NODE__DATA_DIR [node.data_dir]: data
EMQX_NODE__COOKIE [node.cookie]: ******
EMQX_NODE__NAME [node.name]: emqx@192.168.182.245
2024-01-04T06:01:35.949947+00:00 [error] failed_to_check_schema: emqx_conf_schema
2024-01-04T06:01:35.950023+00:00 [error] #{kind => validation_error,matched_type => "authz:file",path => "authorization.sources.3.path",reason => failed_to_read_acl_file,value => "data/authz/acl.conf"}
ERROR: call_hocon_failed: -v -t 2024.01.04.06.01.34 -s emqx_conf_schema -c /opt/emqx/etc/emqx.conf -d /opt/emqx/data/configs generate

What did you expect to happen?

copy acl file data/authz/acl.conf from other Pods

How can we reproduce it (as minimally and precisely as possible)?

edit File Acl Content through dashboard，and restart replicant Pod

Anything else we need to know?

No response

EMQX version

docker image emqx:5.3.2

emqx-operator 2.2.5

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

Log files

[id]

@jiangwensm please help us to investigate this by providing test scenario to reproduce the issue as minimally and precisely as possible.

[jiangwensm]

@jiangwensm please help us to investigate this by providing test scenario to reproduce the issue as minimally and precisely as possible.

@id edit File Acl Content through dashboard，and restart replicant Pod

[jiangwensm]

the docs says:

If file-based authorizer is editted through Dashboard or REST API, EMQX stores the new file to data/authz/acl.conf and stops reading the configuration in the original file.

cluster.hocon(stores the acl file path)：

If a certain cluster node is restarted or some new nodes are added, the node will automatically copy and apply the configuration file cluster.hocon from other nodes within the cluster, therefore there is no need nor recommended to configure it manually.

according to the docs, I guess it copies edited cluster.hocon from other Pods, and won't copy data/authz/acl.conf, thus resulted in this problem.

[id]

Thanks, I managed to reproduce the issue. Verifying with the team.

[id]

@jiangwensm we have identified that this issue is with the emqx-operator.

I'm going to close this issue now since there is no bug in EMQX. Please feel free to follow emqx/emqx-operator#983 instead.