Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

disksup log crashes #9920

Closed
zmstone opened this issue Feb 6, 2023 · 0 comments · Fixed by #9923
Closed

disksup log crashes #9920

zmstone opened this issue Feb 6, 2023 · 0 comments · Fixed by #9923
Labels

Comments

@zmstone
Copy link
Member

zmstone commented Feb 6, 2023

What happened?

2023-02-04T10:10:09.122662+00:00 [error] REPORT_CB/2 CRASH: [{mfa,["gen_server",":","error_info","/7"]},{line,949},{client_info,undefined},{label,{gen_server,terminate}},{last_message,{'EXIT',#Port<0.6>,normal}},{log,[]},{name,disksup},{reason,{port_died,normal}},{state,[{data,[{"OS",{unix,linux}},{"Timeout",1800000},{"Threshold",80},{"DiskData",[{"/dev",1888288,0},{"/run",384248,3},{"/",82001584,34},{"/dev/shm",1921224,1},{"/run/lock",5120,0},{"/sys/fs/cgroup",1921224,0},{"/run/user/1000",384244,0}]}]}]}]; Reason: {error,function_clause,[{gen_server,format_log_multi,[[{mfa,["gen_server",":","error_info","/7"]},{line,949},{client_info,undefined},{label,{gen_server,terminate}},{last_message,{'EXIT',#Port<0.6>,normal}},{log,[]},{name,disksup},{reason,{port_died,normal}},{state,[{data,[{"OS",{unix,linux}},{"Timeout",1800000},{"Threshold",80},{"DiskData",[{"/dev",1888288,0},{"/run",384248,3},{"/",82001584,34},{"/dev/shm",1921224,1},{"/run/lock",5120,0},{"/sys/fs/cgroup",1921224,0},{"/run/user/1000",384244,0}]}]}]}],#{chars_limit => unlimited,depth => 100,encoding => utf8,single_line => true}],[{file,"gen_server.erl"},{line,1074}]},{gen_server,format_log,2,[{file,"gen_server.erl"},{line,1030}]}]}
2023-02-04T10:10:09.128007+00:00 [error] REPORT_CB/2 CRASH: [{mfa,["gen_server",":","error_info","/7"]},{line,949},{client_info,undefined},{label,{gen_server,terminate}},{last_message,{'EXIT',<0.1798.0>,{port_died,normal}}},{log,[]},{name,memsup},{reason,{port_died,normal}},{state,[{data,[{"Timeout",60000}]},{items,{"Memory Usage",[{"Allocated",3467350016},{"Total",3934666752}]}},{items,{"Worst Memory User",[{"Pid",<0.2124.0>},{"Memory",284568}]}}]}]; Reason: {error,function_clause,[{gen_server,format_log_multi,[[{mfa,["gen_server",":","error_info","/7"]},{line,949},{client_info,undefined},{label,{gen_server,terminate}},{last_message,{'EXIT',<0.1798.0>,{port_died,normal}}},{log,[]},{name,memsup},{reason,{port_died,normal}},{state,[{data,[{"Timeout",60000}]},{items,{"Memory Usage",[{"Allocated",3467350016},{"Total",3934666752}]}},{items,{"Worst Memory User",[{"Pid",<0.2124.0>},{"Memory",284568}]}}]}],#{chars_limit => unlimited,depth => 100,encoding => utf8,single_line => true}],[{file,"gen_server.erl"},{line,1074}]},{gen_server,format_log,2,[{file,"gen_server.erl"},{line,1030}]}]}
2023-02-04T10:10:09.128631+00:00 [error] REPORT_CB/2 CRASH: [{mfa,["proc_lib",":","crash_report","/4"]},{line,525},{label,{proc_lib,crash}},{report,[[{initial_call,{memsup,init,['Argument__1']}},{pid,<0.1797.0>},{registered_name,memsup},{error_info,{exit,{port_died,normal},[{gen_server,handle_common_reply,8,[{file,"gen_server.erl"},{line,811}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},{ancestors,[os_mon_sup,<0.1794.0>]},{message_queue_len,0},{messages,[]},{links,[<0.1795.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,10958},{stack_size,29},{reductions,2801015}],[]]}]; Reason: {error,function_clause,[{proc_lib,report_cb,[[{mfa,["proc_lib",":","crash_report","/4"]},{line,525},{label,{proc_lib,crash}},{report,[[{initial_call,{memsup,init,['Argument__1']}},{pid,<0.1797.0>},{registered_name,memsup},{error_info,{exit,{port_died,normal},[{gen_server,handle_common_reply,8,[{file,"gen_server.erl"},{line,811}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},{ancestors,[os_mon_sup,<0.1794.0>]},{message_queue_len,0},{messages,[]},{links,[<0.1795.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,10958},{stack_size,29},{reductions,2801015}],[]]}],#{chars_limit => unlimited,depth => 100,single_line => true}],[{file,"proc_lib.erl"},{line,774}]}]}
2023-02-04T10:10:09.128877+00:00 [error] REPORT_CB/2 CRASH: [{mfa,["supervisor",":","do_restart","/3"]},{line,736},{label,{supervisor,child_terminated}},{report,[{supervisor,{local,os_mon_sup}},{errorContext,child_terminated},{reason,{port_died,normal}},{offender,[{pid,<0.1797.0>},{id,memsup},{mfargs,{memsup,start_link,[]}},{restart_type,permanent},{significant,false},{shutdown,2000},{child_type,worker}]}]}]; Reason: {error,function_clause,[{supervisor,format_log_multi,[[{mfa,["supervisor",":","do_restart","/3"]},{line,736},{label,{supervisor,child_terminated}},{report,[{supervisor,{local,os_mon_sup}},{errorContext,child_terminated},{reason,{port_died,normal}},{offender,[{pid,<0.1797.0>},{id,memsup},{mfargs,{memsup,start_link,[]}},{restart_type,permanent},{significant,false},{shutdown,2000},{child_type,worker}]}]}],#{chars_limit => unlimited,depth => 100,encoding => utf8,single_line => true}],[{file,"supervisor.erl"},{line,1657}]},{supervisor,format_log,2,[{file,"supervisor.erl"},{line,1619}]}]}
2023-02-04T10:10:09.128665+00:00 [error] REPORT_CB/2 CRASH: [{mfa,["proc_lib",":","crash_report","/4"]},{line,525},{label,{proc_lib,crash}},{report,[[{initial_call,{disksup,init,['Argument__1']}},{pid,<0.1796.0>},{registered_name,disksup},{error_info,{exit,{port_died,normal},[{gen_server,handle_common_reply,8,[{file,"gen_server.erl"},{line,811}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},{ancestors,[os_mon_sup,<0.1794.0>]},{message_queue_len,0},{messages,[]},{links,[<0.1795.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,6772},{stack_size,29},{reductions,17706}],[]]}]; Reason: {error,function_clause,[{proc_lib,report_cb,[[{mfa,["proc_lib",":","crash_report","/4"]},{line,525},{label,{proc_lib,crash}},{report,[[{initial_call,{disksup,init,['Argument__1']}},{pid,<0.1796.0>},{registered_name,disksup},{error_info,{exit,{port_died,normal},[{gen_server,handle_common_reply,8,[{file,"gen_server.erl"},{line,811}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},{ancestors,[os_mon_sup,<0.1794.0>]},{message_queue_len,0},{messages,[]},{links,[<0.1795.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,6772},{stack_size,29},{reductions,17706}],[]]}],#{chars_limit => unlimited,depth => 100,single_line => true}],[{file,"proc_lib.erl"},{line,774}]}]}
2023-02-04T10:10:09.129206+00:00 [error] REPORT_CB/2 CRASH: [{mfa,["supervisor",":","do_restart","/3"]},{line,736},{label,{supervisor,child_terminated}},{report,[{supervisor,{local,os_mon_sup}},{errorContext,child_terminated},{reason,{port_died,normal}},{offender,[{pid,<0.1796.0>},{id,disksup},{mfargs,{disksup,start_link,[]}},{restart_type,permanent},{significant,false},{shutdown,2000},{child_type,worker}]}]}]; Reason: {error,function_clause,[{supervisor,format_log_multi,[[{mfa,["supervisor",":","do_restart","/3"]},{line,736},{label,{supervisor,child_terminated}},{report,[{supervisor,{local,os_mon_sup}},{errorContext,child_terminated},{reason,{port_died,normal}},{offender,[{pid,<0.1796.0>},{id,disksup},{mfargs,{disksup,start_link,[]}},{restart_type,permanent},{significant,false},{shutdown,2000},{child_type,worker}]}]}],#{chars_limit => unlimited,depth => 100,encoding => utf8,single_line => true}],[{file,"supervisor.erl"},{line,1657}]},{supervisor,format_log,2,[{file,"supervisor.erl"},{line,1619}]}]}
2023-02-04T10:10:30.521054+00:00 [warning] msg: alarm_is_activated, mfa: emqx_alarm:do_actions/3, line: 416, message: <<"73.34% mem usage">>, name: high_system_memory_usage
2023-02-04T10:11:30.524163+00:00 [warning] msg: alarm_is_activated, mfa: emqx_alarm:do_actions/3, line: 416, message: <<"92.06% cpu usage">>, name: high_cpu_usage
2023-02-04T10:13:30.527804+00:00 [warning] msg: alarm_is_deactivated, mfa: emqx_alarm:do_actions/3, line: 422, name: high_cpu_usage
2023-02-04T10:14:22.978148+00:00 [error] REPORT_CB/2 CRASH: [{mfa,["gen_server",":","error_info","/7"]},{line,949},{client_info,undefined},{label,{gen_server,terminate}},{last_message,{'EXIT',#Port<0.6>,normal}},{log,[]},{name,disksup},{reason,{port_died,normal}},{state,[{data,[{"OS",{unix,linux}},{"Timeout",1800000},{"Threshold",80},{"DiskData",[{"/dev",1888288,0},{"/run",384248,3},{"/",82001584,34},{"/dev/shm",1921224,1},{"/run/lock",5120,0},{"/sys/fs/cgroup",1921224,0},{"/run/user/1000",384244,0}]}]}]}]; Reason: {error,function_clause,[{gen_server,format_log_multi,[[{mfa,["gen_server",":","error_info","/7"]},{line,949},{client_info,undefined},{label,{gen_server,terminate}},{last_message,{'EXIT',#Port<0.6>,normal}},{log,[]},{name,disksup},{reason,{port_died,normal}},{state,[{data,[{"OS",{unix,linux}},{"Timeout",1800000},{"Threshold",80},{"DiskData",[{"/dev",1888288,0},{"/run",384248,3},{"/",82001584,34},{"/dev/shm",1921224,1},{"/run/lock",5120,0},{"/sys/fs/cgroup",1921224,0},{"/run/user/1000",384244,0}]}]}]}],#{chars_limit => unlimited,depth => 100,encoding => utf8,single_line => true}],[{file,"gen_server.erl"},{line,1074}]},{gen_server,format_log,2,[{file,"gen_server.erl"},{line,1030}]}]}
2023-02-04T10:14:22.978432+00:00 [error] REPORT_CB/2 CRASH: [{mfa,["gen_server",":","error_info","/7"]},{line,949},{client_info,undefined},{label,{gen_server,terminate}},{last_message,{'EXIT',<0.1798.0>,{port_died,normal}}},{log,[]},{name,memsup},{reason,{port_died,normal}},{state,[{data,[{"Timeout",60000}]},{items,{"Memory Usage",[{"Allocated",3559731200},{"Total",3934666752}]}},{items,{"Worst Memory User",[{"Pid",<0.1707.0>},{"Memory",4121688}]}}]}]; Reason: {error,function_clause,[{gen_server,format_log_multi,[[{mfa,["gen_server",":","error_info","/7"]},{line,949},{client_info,undefined},{label,{gen_server,terminate}},{last_message,{'EXIT',<0.1798.0>,{port_died,normal}}},{log,[]},{name,memsup},{reason,{port_died,normal}},{state,[{data,[{"Timeout",60000}]},{items,{"Memory Usage",[{"Allocated",3559731200},{"Total",3934666752}]}},{items,{"Worst Memory User",[{"Pid",<0.1707.0>},{"Memory",4121688}]}}]}],#{chars_limit => unlimited,depth => 100,encoding => utf8,single_line => true}],[{file,"gen_server.erl"},{line,1074}]},{gen_server,format_log,2,[{file,"gen_server.erl"},{line,1030}]}]}
2023-02-04T10:14:22.980563+00:00 [error] REPORT_CB/2 CRASH: [{mfa,["proc_lib",":","crash_report","/4"]},{line,525},{label,{proc_lib,crash}},{report,[[{initial_call,{disksup,init,['Argument__1']}},{pid,<0.1796.0>},{registered_name,disksup},{error_info,{exit,{port_died,normal},[{gen_server,handle_common_reply,8,[{file,"gen_server.erl"},{line,811}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},{ancestors,[os_mon_sup,<0.1794.0>]},{message_queue_len,0},{messages,[]},{links,[<0.1795.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,6772},{stack_size,29},{reductions,17630}],[]]}]; Reason: {error,function_clause,[{proc_lib,report_cb,[[{mfa,["proc_lib",":","crash_report","/4"]},{line,525},{label,{proc_lib,crash}},{report,[[{initial_call,{disksup,init,['Argument__1']}},{pid,<0.1796.0>},{registered_name,disksup},{error_info,{exit,{port_died,normal},[{gen_server,handle_common_reply,8,[{file,"gen_server.erl"},{line,811}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},{ancestors,[os_mon_sup,<0.1794.0>]},{message_queue_len,0},{messages,[]},{links,[<0.1795.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,6772},{stack_size,29},{reductions,17630}],[]]}],#{chars_limit => unlimited,depth => 100,single_line => true}],[{file,"proc_lib.erl"},{line,774}]}]}
2023-02-04T10:14:22.980677+00:00 [error] REPORT_CB/2 CRASH: [{mfa,["proc_lib",":","crash_report","/4"]},{line,525},{label,{proc_lib,crash}},{report,[[{initial_call,{memsup,init,['Argument__1']}},{pid,<0.1797.0>},{registered_name,memsup},{error_info,{exit,{port_died,normal},[{gen_server,handle_common_reply,8,[{file,"gen_server.erl"},{line,811}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},{ancestors,[os_mon_sup,<0.1794.0>]},{message_queue_len,0},{messages,[]},{links,[<0.1795.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,6772},{stack_size,29},{reductions,380883}],[]]}]; Reason: {error,function_clause,[{proc_lib,report_cb,[[{mfa,["proc_lib",":","crash_report","/4"]},{line,525},{label,{proc_lib,crash}},{report,[[{initial_call,{memsup,init,['Argument__1']}},{pid,<0.1797.0>},{registered_name,memsup},{error_info,{exit,{port_died,normal},[{gen_server,handle_common_reply,8,[{file,"gen_server.erl"},{line,811}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},{ancestors,[os_mon_sup,<0.1794.0>]},{message_queue_len,0},{messages,[]},{links,[<0.1795.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,6772},{stack_size,29},{reductions,380883}],[]]}],#{chars_limit => unlimited,depth => 100,single_line => true}],[{file,"proc_lib.erl"},{line,774}]}]}
2023-02-04T10:14:22.981115+00:00 [error] REPORT_CB/2 CRASH: [{mfa,["supervisor",":","do_restart","/3"]},{line,736},{label,{supervisor,child_terminated}},{report,[{supervisor,{local,os_mon_sup}},{errorContext,child_terminated},{reason,{port_died,normal}},{offender,[{pid,<0.1796.0>},{id,disksup},{mfargs,{disksup,start_link,[]}},{restart_type,permanent},{significant,false},{shutdown,2000},{child_type,worker}]}]}]; Reason: {error,function_clause,[{supervisor,format_log_multi,[[{mfa,["supervisor",":","do_restart","/3"]},{line,736},{label,{supervisor,child_terminated}},{report,[{supervisor,{local,os_mon_sup}},{errorContext,child_terminated},{reason,{port_died,normal}},{offender,[{pid,<0.1796.0>},{id,disksup},{mfargs,{disksup,start_link,[]}},{restart_type,permanent},{significant,false},{shutdown,2000},{child_type,worker}]}]}],#{chars_limit => unlimited,depth => 100,encoding => utf8,single_line => true}],[{file,"supervisor.erl"},{line,1657}]},{supervisor,format_log,2,[{file,"supervisor.erl"},{line,1619}]}]}
2023-02-04T10:14:22.981488+00:00 [error] REPORT_CB/2 CRASH: [{mfa,["supervisor",":","do_restart","/3"]},{line,736},{label,{supervisor,child_terminated}},{report,[{supervisor,{local,os_mon_sup}},{errorContext,child_terminated},{reason,{port_died,normal}},{offender,[{pid,<0.1797.0>},{id,memsup},{mfargs,{memsup,start_link,[]}},{restart_type,permanent},{significant,false},{shutdown,2000},{child_type,worker}]}]}]; Reason: {error,function_clause,[{supervisor,format_log_multi,[[{mfa,["supervisor",":","do_restart","/3"]},{line,736},{label,{supervisor,child_terminated}},{report,[{supervisor,{local,os_mon_sup}},{errorContext,child_terminated},{reason,{port_died,normal}},{offender,[{pid,<0.1797.0>},{id,memsup},{mfargs,{memsup,start_link,[]}},{restart_type,permanent},{significant,false},{shutdown,2000},{child_type,worker}]}]}],#{chars_limit => unlimited,depth => 100,encoding => utf8,single_line => true}],[{file,"supervisor.erl"},{line,1657}]},{supervisor,format_log,2,[{file,"supervisor.erl"},{line,1619}]}]}
```

### What did you expect to happen?

no crash

### How can we reproduce it (as minimally and precisely as possible)?

_No response_

### Anything else we need to know?

_No response_

### EMQX version

5.0.16

### OS version

see original issue report

### Log files

see description
@zmstone zmstone added the BUG label Feb 6, 2023
@zmstone zmstone changed the title disksup error causes node fail to start disksup log crashes Feb 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant