EMQX crashes at startup if configuration keys are missing

[agronholm]

What happened?

I tried to configure EMQX for authentication over HTTP. I didn't realize I was missing the "mechanism" key and EMQX crashed at startup, giving me a traceback I couldn't decipher.

What did you expect to happen?

I would have expected to see an error message about a missing mandatory configuration parameter.

How can we reproduce it (as minimally and precisely as possible)?

Create the following docker-compose.yml and run docker compose up emqx:

services:
  emqx:
    image: emqx/emqx:5.0.6
    volumes:
      - emqx-data:/opt/emqx/data
      - emqx-etc:/opt/emqx/etc
    tmpfs:
      - /opt/emqx/log
    ports:
      - "127.0.0.1:1883:1883"
      - "127.0.0.1:18083:18083"
    environment:
        EMQX_AUTHENTICATION__1__BACKEND: http
        EMQX_AUTHENTICATION__1__ENABLE: "true"
        EMQX_AUTHENTICATION__1__URL: http://web:8800/v1/integrations/mqtt/authentication

Anything else we need to know?

No response

EMQX version

sysdescr : EMQX version : 5.0.6 datetime : 2022-08-23T14:32:17.491631964+00:00 uptime : 17 minutes, 15 seconds

OS version

Linux fedora 5.18.18-200.fc36.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Aug 17 16:02:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Log files

eb_server-emqx-1 | rpc.port_discovery = EMQX_RPC__PORT_DISCOVERY = manual eb_server-emqx-1 | log.file_handlers.default.enable = EMQX_LOG__FILE_HANDLERS__DEFAULT__ENABLE = false eb_server-emqx-1 | log.console_handler.enable = EMQX_LOG__CONSOLE_HANDLER__ENABLE = true eb_server-emqx-1 | node.name = EMQX_NODE__NAME = emqx@192.168.96.2 eb_server-emqx-1 | 2022-08-23T12:13:24.475890+00:00 [warning] line: 405, message: <<"75.99% mem usage">>, mfa: emqx_alarm:do_actions/3, msg: alarm_is_activated, name: high_system_memory_usage eb_server-emqx-1 | Listener ssl:default on :8883 started. eb_server-emqx-1 | Listener tcp:default on :1883 started. eb_server-emqx-1 | Listener ws:default on :8083 started. eb_server-emqx-1 | Listener wss:default on :8084 started. eb_server-emqx-1 | Listener http:dashboard on :18083 started. eb_server-emqx-1 | 2022-08-23T12:13:24.521792+00:00 [error] crasher: initial call: application_master:init/4, pid: <0.2476.0>, registered_name: [], exit: {{bad_return,{{emqx_authn_app,start,[normal,[]]},{'EXIT',{function_clause,[{emqx_authn,do_check_config,[#{<<"backend">> => <<"http">>,<<"enable">> => true,<<"url">> => "http://web:8800/v1/integrations/mqtt/authentication"},#{}],[{file,"emqx_authn.erl"},{line,58}]},{emqx_authn,check_config,2,[{file,"emqx_authn.erl"},{line,53}]},{emqx_authn,check_configs,1,[{file,"emqx_authn.erl"},{line,47}]},{emqx_authn_app,'-initialize/0-fun-0-',1,[{file,"emqx_authn_app.erl"},{line,56}]},{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},{emqx_authn_app,start,2,[{file,"emqx_authn_app.erl"},{line,40}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}},[{application_master,init,4,[{file,"application_master.erl"},{line,142}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [<0.2475.0>], message_queue_len: 1, messages: [{'EXIT',<0.2477.0>,normal}], links: [<0.2475.0>,<0.1683.0>], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 29, reductions: 212; neighbours: eb_server-emqx-1 | 2022-08-23T12:13:24.525915+00:00 [critical] app: emqx_authn, line: 88, mfa: emqx_machine_boot:start_one_app/1, msg: failed_to_start_app, reason: {emqx_authn,{bad_return,{{emqx_authn_app,start,[normal,[]]},{'EXIT',{function_clause,[{emqx_authn,do_check_config,[#{<<"backend">> => <<"http">>,<<"enable">> => true,<<"url">> => "http://web:8800/v1/integrations/mqtt/authentication"},#{}],[{file,"emqx_authn.erl"},{line,58}]},{emqx_authn,check_config,2,[{file,"emqx_authn.erl"},{line,53}]},{emqx_authn,check_configs,1,[{file,"emqx_authn.erl"},{line,47}]},{emqx_authn_app,'-initialize/0-fun-0-',1,[{file,"emqx_authn_app.erl"},{line,56}]},{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},{emqx_authn_app,start,2,[{file,"emqx_authn_app.erl"},{line,40}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}} eb_server-emqx-1 | 2022-08-23T12:13:24.526123+00:00 [error] Supervisor: {local,emqx_machine_sup}. Context: start_error. Reason: {'EXIT',{{failed_to_start_app,emqx_authn,{emqx_authn,{bad_return,{{emqx_authn_app,start,[normal,[]]},{'EXIT',{function_clause,[{emqx_authn,do_check_config,[#{<<"backend">> => <<"http">>,<<"enable">> => true,<<"url">> => "http://web:8800/v1/integrations/mqtt/authentication"},#{}],[{file,"emqx_authn.erl"},{line,58}]},{emqx_authn,check_config,2,[{file,"emqx_authn.erl"},{line,53}]},{emqx_authn,check_configs,1,[{file,"emqx_authn.erl"},{line,47}]},{emqx_authn_app,'-initialize/0-fun-0-',1,[{file,"emqx_authn_app.erl"},{line,56}]},{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},{emqx_authn_app,start,2,[{file,"emqx_authn_app.erl"},{line,40}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}},[{emqx_machine_boot,start_one_app,1,[{file,"emqx_machine_boot.erl"},{line,89}]},{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},{emqx_machine_boot,post_boot,0,[{file,"emqx_machine_boot.erl"},{line,38}]},{supervisor,do_start_child_i,3,[{file,"supervisor.erl"},{line,414}]},{supervisor,do_start_child,2,[{file,"supervisor.erl"},{line,400}]},{supervisor,'-start_children/2-fun-0-',3,[{file,"supervisor.erl"},{line,384}]},{supervisor,children_map,4,[{file,"supervisor.erl"},{line,1250}]},{supervisor,init_children,2,[{file,"supervisor.erl"},{line,350}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,423}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,390}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}}. Offender: id=emqx_machine_boot,pid=undefined. eb_server-emqx-1 | 2022-08-23T12:13:24.526697+00:00 [error] crasher: initial call: application_master:init/4, pid: <0.1919.0>, registered_name: [], exit: {{{shutdown,{failed_to_start_child,emqx_machine_boot,{'EXIT',{{failed_to_start_app,emqx_authn,{emqx_authn,{bad_return,{{emqx_authn_app,start,[normal,[]]},{'EXIT',{function_clause,[{emqx_authn,do_check_config,[#{<<"backend">> => <<"http">>,<<"enable">> => true,<<"url">> => "http://web:8800/v1/integrations/mqtt/authentication"},#{}],[{file,"emqx_authn.erl"},{line,58}]},{emqx_authn,check_config,2,[{file,"emqx_authn.erl"},{line,53}]},{emqx_authn,check_configs,1,[{file,"emqx_authn.erl"},{line,47}]},{emqx_authn_app,'-initialize/0-fun-0-',1,[{file,"emqx_authn_app.erl"},{line,56}]},{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},{emqx_authn_app,start,2,[{file,"emqx_authn_app.erl"},{line,40}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}},[{emqx_machine_boot,start_one_app,1,[{file,"emqx_machine_boot.erl"},{line,89}]},{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},{emqx_machine_boot,post_boot,0,[{file,"emqx_machine_boot.erl"},{line,38}]},{supervisor,do_start_child_i,3,[{file,"supervisor.erl"},{line,414}]},{supervisor,do_start_child,2,[{file,"supervisor.erl"},{line,400}]},{supervisor,'-start_children/2-fun-0-',3,[{file,"supervisor.erl"},{line,384}]},{supervisor,children_map,4,[{file,"supervisor.erl"},{line,1250}]},{supervisor,init_children,2,[{file,"supervisor.erl"},{line,350}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,423}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,390}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}}}},{emqx_machine_app,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,142}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [<0.1918.0>], message_queue_len: 1, messages: [{'EXIT',<0.1920.0>,normal}], links: [<0.1918.0>,<0.1683.0>], dictionary: [], trap_exit: true, status: running, heap_size: 1598, stack_size: 29, reductions: 162; neighbours: eb_server-emqx-1 | Stop listener http:dashboard on :18083 successfully. eb_server-emqx-1 | Listener ssl:default on :8883 stopped. eb_server-emqx-1 | Listener tcp:default on :1883 stopped. eb_server-emqx-1 | Listener ws:default on :8083 stopped. eb_server-emqx-1 | Listener wss:default on :8084 stopped. eb_server-emqx-1 | [os_mon] memory supervisor port (memsup): Erlang has closed eb_server-emqx-1 | [os_mon] cpu supervisor port (cpu_sup): Erlang has closed eb_server-emqx-1 | {"Kernel pid terminated",application_controller,"{application_start_failure,emqx_machine,{{shutdown,{failed_to_start_child,emqx_machine_boot,{'EXIT',{{failed_to_start_app,emqx_authn,{emqx_authn,{bad_return,{{emqx_authn_app,start,[normal,[]]},{'EXIT',{function_clause,[{emqx_authn,do_check_config,[#{<<\"backend\">> => <<\"http\">>,<<\"enable\">> => true,<<\"url\">> => \"http://web:8800/v1/integrations/mqtt/authentication\"},#{}],[{file,\"emqx_authn.erl\"},{line,58}]},{emqx_authn,check_config,2,[{file,\"emqx_authn.erl\"},{line,53}]},{emqx_authn,check_configs,1,[{file,\"emqx_authn.erl\"},{line,47}]},{emqx_authn_app,'-initialize/0-fun-0-',1,[{file,\"emqx_authn_app.erl\"},{line,56}]},{lists,foreach,2,[{file,\"lists.erl\"},{line,1342}]},{emqx_authn_app,start,2,[{file,\"emqx_authn_app.erl\"},{line,40}]},{application_master,start_it_old,4,[{file,\"application_master.erl\"},{line,293}]}]}}}}}},[{emqx_machine_boot,start_one_app,1,[{file,\"emqx_machine_boot.erl\"},{line,89}]},{lists,foreach,2,[{file,\"lists.erl\"},{line,1342}]},{emqx_machine_boot,post_boot,0,[{file,\"emqx_machine_boot.erl\"},{line,38}]},{supervisor,do_start_child_i,3,[{file,\"supervisor.erl\"},{line,414}]},{supervisor,do_start_child,2,[{file,\"supervisor.erl\"},{line,400}]},{supervisor,'-start_children/2-fun-0-',3,[{file,\"supervisor.erl\"},{line,384}]},{supervisor,children_map,4,[{file,\"supervisor.erl\"},{line,1250}]},{supervisor,init_children,2,[{file,\"supervisor.erl\"},{line,350}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,423}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,390}]},{proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,226}]}]}}}},{emqx_machine_app,start,[normal,[]]}}}"} eb_server-emqx-1 | Kernel pid terminated (application_controller) ({application_start_failure,emqx_machine,{{shutdown,{failed_to_start_child,emqx_machine_boot,{'EXIT',{{failed_to_start_app,emqx_authn,{emqx_authn,{bad_return,{{emqx_authn_app,start,[normal,[]]},{'EXIT',{function_clause,[{emqx_authn,do_check_config,[#{<<"backend">> => <<"http">>,<<"enable">> => true,<<"url">> => "http://web:8800/v1/integrations/mqtt/authentication"},#{}],[{file,"emqx_authn.erl"},{line,58}]},{emqx_authn,check_config,2,[{file,"emqx_authn.erl"},{line,53}]},{emqx_authn,check_configs,1,[{file,"emqx_authn.erl"},{line,47}]},{emqx_authn_app,'-initialize/0-fun-0-',1,[{file,"emqx_authn_app.erl"},{line,56}]},{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},{emqx_authn_app,start,2,[{file,"emqx_authn_app.erl"},{line,40}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,293}]}]}}}}}},[{emqx_machine_boot,start_one_app,1,[{file,"emqx_machine_boot.erl"},{line,89}]},{lists,foreach,2,[{file,"lists.erl"},{line,1342}]},{emqx_machine_boot,p