After installing the plugin in the k8s cluster environment, the repl node cannot be restarted

[Estheriii]

What happened?

After installing the plugin in the k8s cluster environment, the repl node cannot be restarted

What did you expect to happen?

The repl node started successfully

How can we reproduce it (as minimally and precisely as possible)?

1.Start 3core node, 6repl node cluster
2.install plugin
3.Delete one of the repl nodes to automatically restart

Anything else we need to know?

Repl node fails to start properly
An error occurs on the dashboard page:

EMQX version

sysdescr : EMQX version : 5.0.21-g921ce502 datetime : 2023-03-30T09:18:24.112621684+00:00 uptime : 8 minutes, 40 seconds

OS version

# On Linux:
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
# paste output here
Linux emqx-cluster-3 4.19.1-1.el7.elrepo.x86_64 #1 SMP Sun Nov 4 18:04:04 EST 2018 x86_64 GNU/Linux
# paste output here

Log files

[ieQu1]

Hello,

Could you please clarify this part: "3.Delete one of the repl nodes to automatically restart". What do you mean by delete? Remove the pod, or something else?

Also I need full logs from the node that experienced the problem.

[Estheriii]

delete pod in rancher by click delete button on mouse position above
or use 'kubectl delete pod emqx-cluster-0 --force --grace-period=0' to delete pod
logPrintedByRestartedPodAfterRestart.log
otherReplPod.log
otherReplPod-2.log
corePod.log

[Estheriii]

Hello,

Could you please clarify this part: "3.Delete one of the repl nodes to automatically restart". What do you mean by delete? Remove the pod, or something else?

Also I need full logs from the node that experienced the problem.

Is there anything else I can help with? I think this question is easy to reproduce, I don't think it is necessary to use k8s. The problem can be reproduced even in a docker cluster environment or a virtual machine cluster environment. I think the problem is mainly related to the inability to successfully start or find the plug-in when the node starts.

[Estheriii]

hey, there is a problem with the repl node in the form of statefulset, but the problem in the form of deployment no longer exists.

[Estheriii]

When repl is in the deployment environment (using the cluster built by the official operator), errors may also occur. The error occurs when the plugin is installed, started, uninstalled, and reinstalled, resulting in the following error：

2023-04-04T19:01:59.610096+00:00 [error] crasher: initial call: wolff_producer:init/1, pid: <0.32580.1>, registered_name: [], error: {{crc32cer_nif_not_loaded,"plugins/emqx_plugin_kafka-1.0/crc32cer-0.1.8/priv/crc32cer_nif"},[{erlang,nif_error,[{crc32cer_nif_not_loaded,"plugins/emqx_plugin_kafka-1.0/crc32cer-0.1.8/priv/crc32cer_nif"}],[{error_info,#{module => erl_erts_errors}}]},{crc32cer,nif,2,[{file,"/home/zz/work/docker-build/plugin-pod4/base-kafka/_build/default/lib/crc32cer/src/crc32cer.erl"},{line,19}]},{kpro_batch,encode_tx,4,[{file,"/home/zz/work/docker-build/plugin-pod4/base-kafka/_build/default/lib/kafka_protocol/src/kpro_batch.erl"},{line,96}]},{kpro_req_lib,produce,5,[{file,"/home/zz/work/docker-build/plugin-pod4/base-kafka/_build/default/lib/kafka_protocol/src/kpro_req_lib.erl"},{line,286}]},{wolff_producer,send_to_kafka,1,[{file,"/home/zz/work/docker-build/plugin-pod4/base-kafka/_build/default/lib/wolff/src/wolff_producer.erl"},{line,389}]},{wolff_producer,handle_info,2,[{file,"/home/zz/work/docker-build/plugin-pod4/base-kafka/_build/default/lib/wolff/src/wolff_producer.erl"},{line,250}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [emqx_produce,wolff_producers_sup,wolff_sup,<0.31423.1>], message_queue_len: 0, messages: [], links: [<0.31433.1>], dictionary: [{rand_seed,{#{bits => 58,jump => #Fun<rand.3.92093067>,next => #Fun<rand.0.92093067>,type => exsss,uniform => #Fun<rand.1.92093067>,uniform_n => #Fun<rand.2.92093067>},[99084382297699110|207745114345308668]}}], trap_exit: true, status: running, heap_size: 17731, stack_size: 29, reductions: 33009; neighbours:

Then, the plugin will continue to display inactive and cannot be activated normally，And cannot be uninstall.
At this point, restarting a repl node will result in the following error log and the node cannot start

EMQX_RPC__PORT_DISCOVERY [rpc.port_discovery]: manual

EMQX_LOG__FILE_HANDLERS__DEFAULT__ENABLE [log.file_handlers.default.enable]: false

EMQX_LOG__CONSOLE_HANDLER__ENABLE [log.console_handler.enable]: true

EMQX_NODE__DB_ROLE [node.db_role]: replicant

EMQX_NODE__COOKIE [node.cookie]: ******

EMQX_NODE__NAME [node.name]: emqx@10.42.9.221

Listener tcp:default on 0.0.0.0:1883 started.

Listener http:dashboard on :18083 started.

2023-04-04T19:27:02.647571+00:00 [info] msg: =============load conf===================, mfa: authenticate_client_app:load_config/1, line: 68, clientConf: #{<<"client_id">> => "emqx_kafka",<<"connection_strategy">> => per_partition,<<"min_metadata_refresh_interval">> => 5000,<<"query_api_versions">> => true,<<"request_timeout">> => 240000,<<"servers">> => ["emqx-kafka:9092"]}, hooKConf: [#{<<"hook">> => "client.authenticate",<<"key">> => "client_authenticate_key",<<"topic">> => "mqtt-data",<<"value">> => "{\"type\":\"client.authenticate\",\"clientid\" : \"${clientid}\", \"from\" : \"${from}\",\"username\" : \"${username}\",\"topic\" : \"${topic}\", \"payload\" : \"${payload}\", \"qos\" : \"${qos}\",  \"node\" : \"${node}\",\"ts\" : \"${ts}\" }"}], producerConf: #{<<"compression">> => no_compression,<<"encode_payload_type">> => plain,<<"partitioner">> => random,<<"produce">> => async,<<"produce_sync_timeout">> => 3000,<<"replayq_dir">> => <<"undefined">>}

2023-04-04T19:27:02.648907+00:00 [error] crasher: initial call: application_master:init/4, pid: <0.2967.0>, registered_name: [], exit: {{bad_return,{{authenticate_client_app,start,[normal,[]]},{'EXIT',{undef,[{emqx_plugin_kafka_hook,load,[#{<<"client_id">> => "emqx_kafka",<<"connection_strategy">> => per_partition,<<"min_metadata_refresh_interval">> => 5000,<<"query_api_versions">> => true,<<"request_timeout">> => 240000,<<"servers">> => ["emqx-kafka:9092"]},#{<<"compression">> => no_compression,<<"encode_payload_type">> => plain,<<"partitioner">> => random,<<"produce">> => async,<<"produce_sync_timeout">> => 3000,<<"replayq_dir">> => <<"undefined">>},[#{<<"hook">> => "client.authenticate",<<"key">> => "client_authenticate_key",<<"topic">> => "mqtt-data",<<"value">> => "{\"type\":\"client.authenticate\",\"clientid\" : \"${clientid}\", \"from\" : \"${from}\",\"username\" : \"${username}\",\"topic\" : \"${topic}\", \"payload\" : \"${payload}\", \"qos\" : \"${qos}\",  \"node\" : \"${node}\",\"ts\" : \"${ts}\" }"}]],[]},{authenticate_client_app,start,2,[{file,"/home/zz/work/docker-build/plugin-pod4/authenticate-client/src/authenticate_client_app.erl"},{line,45}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}},[{application_master,init,4,[{file,"application_master.erl"},{line,142}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [<0.2966.0>], message_queue_len: 1, messages: [{'EXIT',<0.2968.0>,normal}], links: [<0.2966.0>,<0.1747.0>], dictionary: [], trap_exit: true, status: running, heap_size: 1598, stack_size: 29, reductions: 158; neighbours:

2023-04-04T19:27:02.661145+00:00 [error] crasher: initial call: application_master:init/4, pid: <0.2852.0>, registered_name: [], exit: {{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{"authenticate_client-1.0",#{app => authenticate_client,err_app => authenticate_client,error => "failed_to_start_plugin_app",reason => {bad_return,{{authenticate_client_app,start,[normal,[]]},{'EXIT',{undef,[{emqx_plugin_kafka_hook,load,[#{<<"client_id">> => "emqx_kafka",<<"connection_strategy">> => per_partition,<<"min_metadata_refresh_interval">> => 5000,<<"query_api_versions">> => true,<<"request_timeout">> => 240000,<<"servers">> => ["emqx-kafka:9092"]},#{<<"compression">> => no_compression,<<"encode_payload_type">> => plain,<<"partitioner">> => random,<<"produce">> => async,<<"produce_sync_timeout">> => 3000,<<"replayq_dir">> => <<"undefined">>},[#{<<"hook">> => "client.authenticate",<<"key">> => "client_authenticate_key",<<"topic">> => "mqtt-data",<<"value">> => "{\"type\":\"client.authenticate\",\"clientid\" : \"${clientid}\", \"from\" : \"${from}\",\"username\" : \"${username}\",\"topic\" : \"${topic}\", \"payload\" : \"${payload}\", \"qos\" : \"${qos}\",  \"node\" : \"${node}\",\"ts\" : \"${ts}\" }"}]],[]},{authenticate_client_app,start,2,[{file,"/home/zz/work/docker-build/plugin-pod4/authenticate-client/src/authenticate_client_app.erl"},{line,45}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,"emqx_plugins.erl"},{line,801}]},{emqx_plugins_app,start,2,[{file,"emqx_plugins_app.erl"},{line,28}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}},[{application_master,init,4,[{file,"application_master.erl"},{line,142}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [<0.2851.0>], message_queue_len: 1, messages: [{'EXIT',<0.2853.0>,normal}], links: [<0.2851.0>,<0.1747.0>], dic
tionary: [], trap_exit: true, status: running, heap_size: 1598, stack_size: 29, reductions: 156; neighbours:

2023-04-04T19:27:02.663903+00:00 [critical] msg: failed_to_start_app, mfa: emqx_machine_boot:start_one_app/1, line: 103, app: emqx_plugins, reason: {emqx_plugins,{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{"authenticate_client-1.0",#{app => authenticate_client,err_app => authenticate_client,error => "failed_to_start_plugin_app",reason => {bad_return,{{authenticate_client_app,start,[normal,[]]},{'EXIT',{undef,[{emqx_plugin_kafka_hook,load,[#{<<"client_id">> => "emqx_kafka",<<"connection_strategy">> => per_partition,<<"min_metadata_refresh_interval">> => 5000,<<"query_api_versions">> => true,<<"request_timeout">> => 240000,<<"servers">> => ["emqx-kafka:9092"]},#{<<"compression">> => no_compression,<<"encode_payload_type">> => plain,<<"partitioner">> => random,<<"produce">> => async,<<"produce_sync_timeout">> => 3000,<<"replayq_dir">> => <<"undefined">>},[#{<<"hook">> => "client.authenticate",<<"key">> => "client_authenticate_key",<<"topic">> => "mqtt-data",<<"value">> => "{\"type\":\"client.authenticate\",\"clientid\" : \"${clientid}\", \"from\" : \"${from}\",\"username\" : \"${username}\",\"topic\" : \"${topic}\", \"payload\" : \"${payload}\", \"qos\" : \"${qos}\",  \"node\" : \"${node}\",\"ts\" : \"${ts}\" }"}]],[]},{authenticate_client_app,start,2,[{file,"/home/zz/work/docker-build/plugin-pod4/authenticate-client/src/authenticate_client_app.erl"},{line,45}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,"emqx_plugins.erl"},{line,801}]},{emqx_plugins_app,start,2,[{file,"emqx_plugins_app.erl"},{line,28}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}

2023-04-04T19:27:02.664292+00:00 [error] Supervisor: {local,emqx_machine_sup}. Context: start_error. Reason: {'EXIT',{{failed_to_start_app,emqx_plugins,{emqx_plugins,{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{"authenticate_client-1.0",#{app => authenticate_client,err_app => authenticate_client,error => "failed_to_start_plugin_app",reason => {bad_return,{{authenticate_client_app,start,[normal,[]]},{'EXIT',{undef,[{emqx_plugin_kafka_hook,load,[#{<<"client_id">> => "emqx_kafka",<<"connection_strategy">> => per_partition,<<"min_metadata_refresh_interval">> => 5000,<<"query_api_versions">> => true,<<"request_timeout">> => 240000,<<"servers">> => ["emqx-kafka:9092"]},#{<<"compression">> => no_compression,<<"encode_payload_type">> => plain,<<"partitioner">> => random,<<"produce">> => async,<<"produce_sync_timeout">> => 3000,<<"replayq_dir">> => <<"undefined">>},[#{<<"hook">> => "client.authenticate",<<"key">> => "client_authenticate_key",<<"topic">> => "mqtt-data",<<"value">> => "{\"type\":\"client.authenticate\",\"clientid\" : \"${clientid}\", \"from\" : \"${from}\",\"username\" : \"${username}\",\"topic\" : \"${topic}\", \"payload\" : \"${payload}\", \"qos\" : \"${qos}\",  \"node\" : \"${node}\",\"ts\" : \"${ts}\" }"}]],[]},{authenticate_client_app,start,2,[{file,"/home/zz/work/docker-build/plugin-pod4/authenticate-client/src/authenticate_client_app.erl"},{line,45}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,"emqx_plugins.erl"},{line,801}]},{emqx_plugins_app,start,2,[{file,"emqx_plugins_app.erl"},{line,28}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}},[{emqx_machine_boot,start_one_app,1,[{file,"emqx_machine_boot.erl"},{line,104}]},{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},{emqx_machine_boot,post_boot,0,[{file,"emqx_machine_boot.erl"},{line,39}]},{supervisor,do_start_chi
ld_i,3,[{file,"supervisor.erl"},{line,414}]},{supervisor,do_start_child,2,[{file,"supervisor.erl"},{line,400}]},{supervisor,'-start_children/2-fun-0-',3,[{file,"supervisor.erl"},{line,384}]},{supervisor,children_map,4,[{file,"supervisor.erl"},{line,1250}]},{supervisor,init_children,2,[{file,"supervisor.erl"},{line,350}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,423}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,390}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}}. Offender: id=emqx_machine_boot,pid=undefined.

2023-04-04T19:27:02.665347+00:00 [error] crasher: initial call: application_master:init/4, pid: <0.2024.0>, registered_name: [], exit: {{{shutdown,{failed_to_start_child,emqx_machine_boot,{'EXIT',{{failed_to_start_app,emqx_plugins,{emqx_plugins,{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{"authenticate_client-1.0",#{app => authenticate_client,err_app => authenticate_client,error => "failed_to_start_plugin_app",reason => {bad_return,{{authenticate_client_app,start,[normal,[]]},{'EXIT',{undef,[{emqx_plugin_kafka_hook,load,[#{<<"client_id">> => "emqx_kafka",<<"connection_strategy">> => per_partition,<<"min_metadata_refresh_interval">> => 5000,<<"query_api_versions">> => true,<<"request_timeout">> => 240000,<<"servers">> => ["emqx-kafka:9092"]},#{<<"compression">> => no_compression,<<"encode_payload_type">> => plain,<<"partitioner">> => random,<<"produce">> => async,<<"produce_sync_timeout">> => 3000,<<"replayq_dir">> => <<"undefined">>},[#{<<"hook">> => "client.authenticate",<<"key">> => "client_authenticate_key",<<"topic">> => "mqtt-data",<<"value">> => "{\"type\":\"client.authenticate\",\"clientid\" : \"${clientid}\", \"from\" : \"${from}\",\"username\" : \"${username}\",\"topic\" : \"${topic}\", \"payload\" : \"${payload}\", \"qos\" : \"${qos}\",  \"node\" : \"${node}\",\"ts\" : \"${ts}\" }"}]],[]},{authenticate_client_app,start,2,[{file,"/home/zz/work/docker-build/plugin-pod4/authenticate-client/src/authenticate_client_app.erl"},{line,45}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,"emqx_plugins.erl"},{line,801}]},{emqx_plugins_app,start,2,[{file,"emqx_plugins_app.erl"},{line,28}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}},[{emqx_machine_boot,start_one_app,1,[{file,"emqx_machine_boot.erl"},{line,104}]},{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},{emqx_machine_boot,p
ost_boot,0,[{file,"emqx_machine_boot.erl"},{line,39}]},{supervisor,do_start_child_i,3,[{file,"supervisor.erl"},{line,414}]},{supervisor,do_start_child,2,[{file,"supervisor.erl"},{line,400}]},{supervisor,'-start_children/2-fun-0-',3,[{file,"supervisor.erl"},{line,384}]},{supervisor,children_map,4,[{file,"supervisor.erl"},{line,1250}]},{supervisor,init_children,2,[{file,"supervisor.erl"},{line,350}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,423}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,390}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}}}},{emqx_machine_app,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,142}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [<0.2023.0>], message_queue_len: 1, messages: [{'EXIT',<0.2025.0>,normal}], links: [<0.2023.0>,<0.1747.0>], dictionary: [], trap_exit: true, status: running, heap_size: 2586, stack_size: 29, reductions: 156; neighbours:

Stop listener http:dashboard on :18083 successfully.

Listener tcp:default on 0.0.0.0:1883 stopped.

[os_mon] memory supervisor port (memsup): Erlang has closed

[os_mon] cpu supervisor port (cpu_sup): Erlang has closed

{"Kernel pid terminated",application_controller,"{application_start_failure,emqx_machine,{{shutdown,{failed_to_start_child,emqx_machine_boot,{'EXIT',{{failed_to_start_app,emqx_plugins,{emqx_plugins,{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{\"authenticate_client-1.0\",#{app => authenticate_client,err_app => authenticate_client,error => \"failed_to_start_plugin_app\",reason => {bad_return,{{authenticate_client_app,start,[normal,[]]},{'EXIT',{undef,[{emqx_plugin_kafka_hook,load,[#{<<\"client_id\">> => \"emqx_kafka\",<<\"connection_strategy\">> => per_partition,<<\"min_metadata_refresh_interval\">> => 5000,<<\"query_api_versions\">> => true,<<\"request_timeout\">> => 240000,<<\"servers\">> => [\"emqx-kafka:9092\"]},#{<<\"compression\">> => no_compression,<<\"encode_payload_type\">> => plain,<<\"partitioner\">> => random,<<\"produce\">> => async,<<\"produce_sync_timeout\">> => 3000,<<\"replayq_dir\">> => <<\"undefined\">>},[#{<<\"hook\">> => \"client.authenticate\",<<\"key\">> => \"client_authenticate_key\",<<\"topic\">> => \"mqtt-data\",<<\"value\">> => \"{\\"type\\":\\"client.authenticate\\",\\"clientid\\" : \\"${clientid}\\", \\"from\\" : \\"${from}\\",\\"username\\" : \\"${username}\\",\\"topic\\" : \\"${topic}\\", \\"payload\\" : \\"${payload}\\", \\"qos\\" : \\"${qos}\\",  \\"node\\" : \\"${node}\\",\\"ts\\" : \\"${ts}\\" }\"}]],[]},{authenticate_client_app,start,2,[{file,\"/home/zz/work/docker-build/plugin-pod4/authenticate-client/src/authenticate_client_app.erl\"},{line,45}]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,293}]}]}}}}}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,\"emqx_plugins.erl\"},{line,801}]},{emqx_plugins_app,start,2,[{file,\"emqx_plugins_app.erl\"},{line,28}]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,293}]}]}}}}}},[{emqx_machine_boot,start_one_app,1,[{file,\"emqx_machine_boot.erl\"},{line,104}]},{lists,foreach,2,[{f
ile,\"lists.erl\"},{line,1342}]},{emqx_machine_boot,post_boot,0,[{file,\"emqx_machine_boot.erl\"},{line,39}]},{supervisor,do_start_child_i,3,[{file,\"supervisor.erl\"},{line,414}]},{supervisor,do_start_child,2,[{file,\"supervisor.erl\"},{line,400}]},{supervisor,'-start_children/2-fun-0-',3,[{file,\"supervisor.erl\"},{line,384}]},{supervisor,children_map,4,[{file,\"supervisor.erl\"},{line,1250}]},{supervisor,init_children,2,[{file,\"supervisor.erl\"},{line,350}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,423}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,390}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,226}]}]}}}},{emqx_machine_app,start,[normal,[]]}}}"}

Kernel pid terminated (application_controller) ({application_start_failure,emqx_machine,{{shutdown,{failed_to_start_child,emqx_machine_boot,{'EXIT',{{failed_to_start_app,emqx_plugins,{emqx_plugins,{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{"authenticate_client-1.0",#{app => authenticate_client,err_app => authenticate_client,error => "failed_to_start_plugin_app",reason => {bad_return,{{authenticate_client_app,start,[normal,[]]},{'EXIT',{undef,[{emqx_plugin_kafka_hook,load,[#{<<"client_id">> => "emqx_kafka",<<"connection_strategy">> => per_partition,<<"min_metadata_refresh_interval">> => 5000,<<"query_api_versions">> => true,<<"request_timeout">> => 240000,<<"servers">> => ["emqx-kafka:9092"]},#{<<"compression">> => no_compression,<<"encode_payload_type">> => plain,<<"partitioner">> => random,<<"produce">> => async,<<"produce_sync_timeout">> => 3000,<<"replayq_dir">> => <<"undefined">>},[#{<<"hook">> => "client.authenticate",<<"key">> => "client_authenticate_key",<<"topic">> => "mqtt

Crash dump is being written to: log/erl_crash.dump...done

[Estheriii]

When repl is in the deployment environment (using the cluster built by the official operator), errors may also occur. The error occurs when the plugin is installed, started, uninstalled, and reinstalled, resulting in the following error：

2023-04-04T19:01:59.610096+00:00 [error] crasher: initial call: wolff_producer:init/1, pid: <0.32580.1>, registered_name: [], error: {{crc32cer_nif_not_loaded,"plugins/emqx_plugin_kafka-1.0/crc32cer-0.1.8/priv/crc32cer_nif"},[{erlang,nif_error,[{crc32cer_nif_not_loaded,"plugins/emqx_plugin_kafka-1.0/crc32cer-0.1.8/priv/crc32cer_nif"}],[{error_info,#{module => erl_erts_errors}}]},{crc32cer,nif,2,[{file,"/home/zz/work/docker-build/plugin-pod4/base-kafka/_build/default/lib/crc32cer/src/crc32cer.erl"},{line,19}]},{kpro_batch,encode_tx,4,[{file,"/home/zz/work/docker-build/plugin-pod4/base-kafka/_build/default/lib/kafka_protocol/src/kpro_batch.erl"},{line,96}]},{kpro_req_lib,produce,5,[{file,"/home/zz/work/docker-build/plugin-pod4/base-kafka/_build/default/lib/kafka_protocol/src/kpro_req_lib.erl"},{line,286}]},{wolff_producer,send_to_kafka,1,[{file,"/home/zz/work/docker-build/plugin-pod4/base-kafka/_build/default/lib/wolff/src/wolff_producer.erl"},{line,389}]},{wolff_producer,handle_info,2,[{file,"/home/zz/work/docker-build/plugin-pod4/base-kafka/_build/default/lib/wolff/src/wolff_producer.erl"},{line,250}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,695}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,771}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [emqx_produce,wolff_producers_sup,wolff_sup,<0.31423.1>], message_queue_len: 0, messages: [], links: [<0.31433.1>], dictionary: [{rand_seed,{#{bits => 58,jump => #Fun<rand.3.92093067>,next => #Fun<rand.0.92093067>,type => exsss,uniform => #Fun<rand.1.92093067>,uniform_n => #Fun<rand.2.92093067>},[99084382297699110|207745114345308668]}}], trap_exit: true, status: running, heap_size: 17731, stack_size: 29, reductions: 33009; neighbours:

Then, the plugin will continue to display inactive and cannot be activated normally，And cannot be uninstall. At this point, restarting a repl node will result in the following error log and the node cannot start

EMQX_RPC__PORT_DISCOVERY [rpc.port_discovery]: manual

EMQX_LOG__FILE_HANDLERS__DEFAULT__ENABLE [log.file_handlers.default.enable]: false

EMQX_LOG__CONSOLE_HANDLER__ENABLE [log.console_handler.enable]: true

EMQX_NODE__DB_ROLE [node.db_role]: replicant

EMQX_NODE__COOKIE [node.cookie]: ******

EMQX_NODE__NAME [node.name]: emqx@10.42.9.221

Listener tcp:default on 0.0.0.0:1883 started.

Listener http:dashboard on :18083 started.

2023-04-04T19:27:02.647571+00:00 [info] msg: =============load conf===================, mfa: authenticate_client_app:load_config/1, line: 68, clientConf: #{<<"client_id">> => "emqx_kafka",<<"connection_strategy">> => per_partition,<<"min_metadata_refresh_interval">> => 5000,<<"query_api_versions">> => true,<<"request_timeout">> => 240000,<<"servers">> => ["emqx-kafka:9092"]}, hooKConf: [#{<<"hook">> => "client.authenticate",<<"key">> => "client_authenticate_key",<<"topic">> => "mqtt-data",<<"value">> => "{\"type\":\"client.authenticate\",\"clientid\" : \"${clientid}\", \"from\" : \"${from}\",\"username\" : \"${username}\",\"topic\" : \"${topic}\", \"payload\" : \"${payload}\", \"qos\" : \"${qos}\",  \"node\" : \"${node}\",\"ts\" : \"${ts}\" }"}], producerConf: #{<<"compression">> => no_compression,<<"encode_payload_type">> => plain,<<"partitioner">> => random,<<"produce">> => async,<<"produce_sync_timeout">> => 3000,<<"replayq_dir">> => <<"undefined">>}

2023-04-04T19:27:02.648907+00:00 [error] crasher: initial call: application_master:init/4, pid: <0.2967.0>, registered_name: [], exit: {{bad_return,{{authenticate_client_app,start,[normal,[]]},{'EXIT',{undef,[{emqx_plugin_kafka_hook,load,[#{<<"client_id">> => "emqx_kafka",<<"connection_strategy">> => per_partition,<<"min_metadata_refresh_interval">> => 5000,<<"query_api_versions">> => true,<<"request_timeout">> => 240000,<<"servers">> => ["emqx-kafka:9092"]},#{<<"compression">> => no_compression,<<"encode_payload_type">> => plain,<<"partitioner">> => random,<<"produce">> => async,<<"produce_sync_timeout">> => 3000,<<"replayq_dir">> => <<"undefined">>},[#{<<"hook">> => "client.authenticate",<<"key">> => "client_authenticate_key",<<"topic">> => "mqtt-data",<<"value">> => "{\"type\":\"client.authenticate\",\"clientid\" : \"${clientid}\", \"from\" : \"${from}\",\"username\" : \"${username}\",\"topic\" : \"${topic}\", \"payload\" : \"${payload}\", \"qos\" : \"${qos}\",  \"node\" : \"${node}\",\"ts\" : \"${ts}\" }"}]],[]},{authenticate_client_app,start,2,[{file,"/home/zz/work/docker-build/plugin-pod4/authenticate-client/src/authenticate_client_app.erl"},{line,45}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}},[{application_master,init,4,[{file,"application_master.erl"},{line,142}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [<0.2966.0>], message_queue_len: 1, messages: [{'EXIT',<0.2968.0>,normal}], links: [<0.2966.0>,<0.1747.0>], dictionary: [], trap_exit: true, status: running, heap_size: 1598, stack_size: 29, reductions: 158; neighbours:

2023-04-04T19:27:02.661145+00:00 [error] crasher: initial call: application_master:init/4, pid: <0.2852.0>, registered_name: [], exit: {{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{"authenticate_client-1.0",#{app => authenticate_client,err_app => authenticate_client,error => "failed_to_start_plugin_app",reason => {bad_return,{{authenticate_client_app,start,[normal,[]]},{'EXIT',{undef,[{emqx_plugin_kafka_hook,load,[#{<<"client_id">> => "emqx_kafka",<<"connection_strategy">> => per_partition,<<"min_metadata_refresh_interval">> => 5000,<<"query_api_versions">> => true,<<"request_timeout">> => 240000,<<"servers">> => ["emqx-kafka:9092"]},#{<<"compression">> => no_compression,<<"encode_payload_type">> => plain,<<"partitioner">> => random,<<"produce">> => async,<<"produce_sync_timeout">> => 3000,<<"replayq_dir">> => <<"undefined">>},[#{<<"hook">> => "client.authenticate",<<"key">> => "client_authenticate_key",<<"topic">> => "mqtt-data",<<"value">> => "{\"type\":\"client.authenticate\",\"clientid\" : \"${clientid}\", \"from\" : \"${from}\",\"username\" : \"${username}\",\"topic\" : \"${topic}\", \"payload\" : \"${payload}\", \"qos\" : \"${qos}\",  \"node\" : \"${node}\",\"ts\" : \"${ts}\" }"}]],[]},{authenticate_client_app,start,2,[{file,"/home/zz/work/docker-build/plugin-pod4/authenticate-client/src/authenticate_client_app.erl"},{line,45}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,"emqx_plugins.erl"},{line,801}]},{emqx_plugins_app,start,2,[{file,"emqx_plugins_app.erl"},{line,28}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}},[{application_master,init,4,[{file,"application_master.erl"},{line,142}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [<0.2851.0>], message_queue_len: 1, messages: [{'EXIT',<0.2853.0>,normal}], links: [<0.2851.0>,<0.1747.0>], dic
tionary: [], trap_exit: true, status: running, heap_size: 1598, stack_size: 29, reductions: 156; neighbours:

2023-04-04T19:27:02.663903+00:00 [critical] msg: failed_to_start_app, mfa: emqx_machine_boot:start_one_app/1, line: 103, app: emqx_plugins, reason: {emqx_plugins,{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{"authenticate_client-1.0",#{app => authenticate_client,err_app => authenticate_client,error => "failed_to_start_plugin_app",reason => {bad_return,{{authenticate_client_app,start,[normal,[]]},{'EXIT',{undef,[{emqx_plugin_kafka_hook,load,[#{<<"client_id">> => "emqx_kafka",<<"connection_strategy">> => per_partition,<<"min_metadata_refresh_interval">> => 5000,<<"query_api_versions">> => true,<<"request_timeout">> => 240000,<<"servers">> => ["emqx-kafka:9092"]},#{<<"compression">> => no_compression,<<"encode_payload_type">> => plain,<<"partitioner">> => random,<<"produce">> => async,<<"produce_sync_timeout">> => 3000,<<"replayq_dir">> => <<"undefined">>},[#{<<"hook">> => "client.authenticate",<<"key">> => "client_authenticate_key",<<"topic">> => "mqtt-data",<<"value">> => "{\"type\":\"client.authenticate\",\"clientid\" : \"${clientid}\", \"from\" : \"${from}\",\"username\" : \"${username}\",\"topic\" : \"${topic}\", \"payload\" : \"${payload}\", \"qos\" : \"${qos}\",  \"node\" : \"${node}\",\"ts\" : \"${ts}\" }"}]],[]},{authenticate_client_app,start,2,[{file,"/home/zz/work/docker-build/plugin-pod4/authenticate-client/src/authenticate_client_app.erl"},{line,45}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,"emqx_plugins.erl"},{line,801}]},{emqx_plugins_app,start,2,[{file,"emqx_plugins_app.erl"},{line,28}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}

2023-04-04T19:27:02.664292+00:00 [error] Supervisor: {local,emqx_machine_sup}. Context: start_error. Reason: {'EXIT',{{failed_to_start_app,emqx_plugins,{emqx_plugins,{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{"authenticate_client-1.0",#{app => authenticate_client,err_app => authenticate_client,error => "failed_to_start_plugin_app",reason => {bad_return,{{authenticate_client_app,start,[normal,[]]},{'EXIT',{undef,[{emqx_plugin_kafka_hook,load,[#{<<"client_id">> => "emqx_kafka",<<"connection_strategy">> => per_partition,<<"min_metadata_refresh_interval">> => 5000,<<"query_api_versions">> => true,<<"request_timeout">> => 240000,<<"servers">> => ["emqx-kafka:9092"]},#{<<"compression">> => no_compression,<<"encode_payload_type">> => plain,<<"partitioner">> => random,<<"produce">> => async,<<"produce_sync_timeout">> => 3000,<<"replayq_dir">> => <<"undefined">>},[#{<<"hook">> => "client.authenticate",<<"key">> => "client_authenticate_key",<<"topic">> => "mqtt-data",<<"value">> => "{\"type\":\"client.authenticate\",\"clientid\" : \"${clientid}\", \"from\" : \"${from}\",\"username\" : \"${username}\",\"topic\" : \"${topic}\", \"payload\" : \"${payload}\", \"qos\" : \"${qos}\",  \"node\" : \"${node}\",\"ts\" : \"${ts}\" }"}]],[]},{authenticate_client_app,start,2,[{file,"/home/zz/work/docker-build/plugin-pod4/authenticate-client/src/authenticate_client_app.erl"},{line,45}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,"emqx_plugins.erl"},{line,801}]},{emqx_plugins_app,start,2,[{file,"emqx_plugins_app.erl"},{line,28}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}},[{emqx_machine_boot,start_one_app,1,[{file,"emqx_machine_boot.erl"},{line,104}]},{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},{emqx_machine_boot,post_boot,0,[{file,"emqx_machine_boot.erl"},{line,39}]},{supervisor,do_start_chi
ld_i,3,[{file,"supervisor.erl"},{line,414}]},{supervisor,do_start_child,2,[{file,"supervisor.erl"},{line,400}]},{supervisor,'-start_children/2-fun-0-',3,[{file,"supervisor.erl"},{line,384}]},{supervisor,children_map,4,[{file,"supervisor.erl"},{line,1250}]},{supervisor,init_children,2,[{file,"supervisor.erl"},{line,350}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,423}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,390}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}}. Offender: id=emqx_machine_boot,pid=undefined.

2023-04-04T19:27:02.665347+00:00 [error] crasher: initial call: application_master:init/4, pid: <0.2024.0>, registered_name: [], exit: {{{shutdown,{failed_to_start_child,emqx_machine_boot,{'EXIT',{{failed_to_start_app,emqx_plugins,{emqx_plugins,{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{"authenticate_client-1.0",#{app => authenticate_client,err_app => authenticate_client,error => "failed_to_start_plugin_app",reason => {bad_return,{{authenticate_client_app,start,[normal,[]]},{'EXIT',{undef,[{emqx_plugin_kafka_hook,load,[#{<<"client_id">> => "emqx_kafka",<<"connection_strategy">> => per_partition,<<"min_metadata_refresh_interval">> => 5000,<<"query_api_versions">> => true,<<"request_timeout">> => 240000,<<"servers">> => ["emqx-kafka:9092"]},#{<<"compression">> => no_compression,<<"encode_payload_type">> => plain,<<"partitioner">> => random,<<"produce">> => async,<<"produce_sync_timeout">> => 3000,<<"replayq_dir">> => <<"undefined">>},[#{<<"hook">> => "client.authenticate",<<"key">> => "client_authenticate_key",<<"topic">> => "mqtt-data",<<"value">> => "{\"type\":\"client.authenticate\",\"clientid\" : \"${clientid}\", \"from\" : \"${from}\",\"username\" : \"${username}\",\"topic\" : \"${topic}\", \"payload\" : \"${payload}\", \"qos\" : \"${qos}\",  \"node\" : \"${node}\",\"ts\" : \"${ts}\" }"}]],[]},{authenticate_client_app,start,2,[{file,"/home/zz/work/docker-build/plugin-pod4/authenticate-client/src/authenticate_client_app.erl"},{line,45}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,"emqx_plugins.erl"},{line,801}]},{emqx_plugins_app,start,2,[{file,"emqx_plugins_app.erl"},{line,28}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}},[{emqx_machine_boot,start_one_app,1,[{file,"emqx_machine_boot.erl"},{line,104}]},{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},{emqx_machine_boot,p
ost_boot,0,[{file,"emqx_machine_boot.erl"},{line,39}]},{supervisor,do_start_child_i,3,[{file,"supervisor.erl"},{line,414}]},{supervisor,do_start_child,2,[{file,"supervisor.erl"},{line,400}]},{supervisor,'-start_children/2-fun-0-',3,[{file,"supervisor.erl"},{line,384}]},{supervisor,children_map,4,[{file,"supervisor.erl"},{line,1250}]},{supervisor,init_children,2,[{file,"supervisor.erl"},{line,350}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,423}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,390}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}}}},{emqx_machine_app,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,142}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [<0.2023.0>], message_queue_len: 1, messages: [{'EXIT',<0.2025.0>,normal}], links: [<0.2023.0>,<0.1747.0>], dictionary: [], trap_exit: true, status: running, heap_size: 2586, stack_size: 29, reductions: 156; neighbours:

Stop listener http:dashboard on :18083 successfully.

Listener tcp:default on 0.0.0.0:1883 stopped.

[os_mon] memory supervisor port (memsup): Erlang has closed

[os_mon] cpu supervisor port (cpu_sup): Erlang has closed

{"Kernel pid terminated",application_controller,"{application_start_failure,emqx_machine,{{shutdown,{failed_to_start_child,emqx_machine_boot,{'EXIT',{{failed_to_start_app,emqx_plugins,{emqx_plugins,{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{\"authenticate_client-1.0\",#{app => authenticate_client,err_app => authenticate_client,error => \"failed_to_start_plugin_app\",reason => {bad_return,{{authenticate_client_app,start,[normal,[]]},{'EXIT',{undef,[{emqx_plugin_kafka_hook,load,[#{<<\"client_id\">> => \"emqx_kafka\",<<\"connection_strategy\">> => per_partition,<<\"min_metadata_refresh_interval\">> => 5000,<<\"query_api_versions\">> => true,<<\"request_timeout\">> => 240000,<<\"servers\">> => [\"emqx-kafka:9092\"]},#{<<\"compression\">> => no_compression,<<\"encode_payload_type\">> => plain,<<\"partitioner\">> => random,<<\"produce\">> => async,<<\"produce_sync_timeout\">> => 3000,<<\"replayq_dir\">> => <<\"undefined\">>},[#{<<\"hook\">> => \"client.authenticate\",<<\"key\">> => \"client_authenticate_key\",<<\"topic\">> => \"mqtt-data\",<<\"value\">> => \"{\\"type\\":\\"client.authenticate\\",\\"clientid\\" : \\"${clientid}\\", \\"from\\" : \\"${from}\\",\\"username\\" : \\"${username}\\",\\"topic\\" : \\"${topic}\\", \\"payload\\" : \\"${payload}\\", \\"qos\\" : \\"${qos}\\",  \\"node\\" : \\"${node}\\",\\"ts\\" : \\"${ts}\\" }\"}]],[]},{authenticate_client_app,start,2,[{file,\"/home/zz/work/docker-build/plugin-pod4/authenticate-client/src/authenticate_client_app.erl\"},{line,45}]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,293}]}]}}}}}}],function => fun emqx_plugins:do_ensure_started/1},[{emqx_plugins,for_plugins,1,[{file,\"emqx_plugins.erl\"},{line,801}]},{emqx_plugins_app,start,2,[{file,\"emqx_plugins_app.erl\"},{line,28}]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,293}]}]}}}}}},[{emqx_machine_boot,start_one_app,1,[{file,\"emqx_machine_boot.erl\"},{line,104}]},{lists,foreach,2,[{f
ile,\"lists.erl\"},{line,1342}]},{emqx_machine_boot,post_boot,0,[{file,\"emqx_machine_boot.erl\"},{line,39}]},{supervisor,do_start_child_i,3,[{file,\"supervisor.erl\"},{line,414}]},{supervisor,do_start_child,2,[{file,\"supervisor.erl\"},{line,400}]},{supervisor,'-start_children/2-fun-0-',3,[{file,\"supervisor.erl\"},{line,384}]},{supervisor,children_map,4,[{file,\"supervisor.erl\"},{line,1250}]},{supervisor,init_children,2,[{file,\"supervisor.erl\"},{line,350}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,423}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,390}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,226}]}]}}}},{emqx_machine_app,start,[normal,[]]}}}"}

Kernel pid terminated (application_controller) ({application_start_failure,emqx_machine,{{shutdown,{failed_to_start_child,emqx_machine_boot,{'EXIT',{{failed_to_start_app,emqx_plugins,{emqx_plugins,{bad_return,{{emqx_plugins_app,start,[normal,[]]},{'EXIT',{#{errors => [{"authenticate_client-1.0",#{app => authenticate_client,err_app => authenticate_client,error => "failed_to_start_plugin_app",reason => {bad_return,{{authenticate_client_app,start,[normal,[]]},{'EXIT',{undef,[{emqx_plugin_kafka_hook,load,[#{<<"client_id">> => "emqx_kafka",<<"connection_strategy">> => per_partition,<<"min_metadata_refresh_interval">> => 5000,<<"query_api_versions">> => true,<<"request_timeout">> => 240000,<<"servers">> => ["emqx-kafka:9092"]},#{<<"compression">> => no_compression,<<"encode_payload_type">> => plain,<<"partitioner">> => random,<<"produce">> => async,<<"produce_sync_timeout">> => 3000,<<"replayq_dir">> => <<"undefined">>},[#{<<"hook">> => "client.authenticate",<<"key">> => "client_authenticate_key",<<"topic">> => "mqtt

Crash dump is being written to: log/erl_crash.dump...done

@zmstone hi, I think the problem is related to crc32cer and emqx_pluginrel