Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broker crash when forwarding message in cluster #2290

Closed
h00s opened this issue Mar 8, 2019 · 5 comments
Assignees
Labels
Milestone

Comments

@h00s
Copy link

@h00s h00s commented Mar 8, 2019

Environment

  • OS: Linux 4.9.0-8-amd64 #1 SMP Debian 4.9.144-3 (2019-02-02) x86_64 GNU/Linux
  • Erlang/OTP: 21.2
  • EMQ: v3.1-beta1

Description

Two brokers running v3.1-beta1. Joining one broker to another in cluster starts with no problems and no crash is occurring.

Then, I can start OwnTracks on phone, send message to one node, and immediately said node is starting to crash. This is the error:

2019-03-08 10:01:35.293 [error] event=authentication_reception_failed socket="#Port<0.90>" reason="timeout"
2019-03-08 10:01:39.304 [error] event=client_authentication_failed driver=tcp reason="{badtcp,timeout}"
2019-03-08 10:01:39.307 [error] h00s@:57968 ** Generic server <0.1899.0> terminating
** Last message in was {tcp,#Port<0.86>, }
** When Server state == {state,esockd_transport,#Port<0.86>,
{{},57968},
undefined,running,100,
{pstate,external,
#Fun<emqx_connection.0.118827427>,
{{},57968},
nossl,3,<<"MQIsdp">>,<<"h00s">>,false,
<0.1899.0>,undefined,undefined,<<"h00s">>,
<0.1782.0>,false,#{},1048576,900,undefined,
false,false,true,true,ignore,
#{msg => 14,pkt => 31},
#{msg => 15,pkt => 31},
true,
{1552,35277,975439},
false,
#{from_client => 0,to_client => 0}},
{none,#{max_packet_size => 1048576,version => 3}},
{emqx_gc,
#{cnt => {1000,997},oct => {1048576,1048299}}},
{keepalive,#Fun<emqx_connection.7.118827427>,103,
675,
{keepalive,check},
#Ref<0.3169275439.1433665537.59751>,0},
true,#Ref<0.3169275439.1433665537.66317>,
undefined,undefined,undefined,15000}
** Reason for termination ==
** {function_clause,[{emqx_broker,dispatch,
[<<"owntracks/+/+">>,{badtcp,timeout}],
[{file,"src/emqx_broker.erl"},{line,239}]},
{lists,foldl,3,[{file,"lists.erl"},{line,1263}]},
{emqx_broker,publish,1,
[{file,"src/emqx_broker.erl"},{line,172}]},
{emqx_protocol,do_publish,2,
[{file,"src/emqx_protocol.erl"},
{line,557}]},
{emqx_connection,handle_packet,2,
[{file,"src/emqx_connection.erl"},
{line,352}]},
{emqx_connection,process_incoming,2,
[{file,"src/emqx_connection.erl"},
{line,332}]},
{gen_server,try_dispatch,4,
[{file,"gen_server.erl"},{line,637}]},
{gen_server,handle_msg,6,
[{file,"gen_server.erl"},{line,711}]}]}
2019-03-08 10:01:54.367 [error] event=authentication_reception_failed socket="#Port<0.91>" reason="timeout"
2019-03-08 10:01:55.024 [error] h00s@:57988 [Broker] Failed to forward msg to emqx@: timeout
2019-03-08 10:01:58.376 [error] event=client_authentication_failed driver=tcp reason="{badtcp,timeout}"
2019-03-08 10:01:58.382 [error] h00s@:57988 [Broker] Failed to forward msg to emqx@: {unknown_error,{{badtcp,timeout},
{gen_server,call,
[<0.2034.0>,
{{call,emqx_broker,
dispatch,
[<<"owntracks/h00s/kruno">>,
{delivery,
<0.2035.0>,
{message,
<<>>,
1,
<<"h00s">>,
#{dup => true,
retain =>
true},
#{username =>
<<"h00s">>},
<<"owntracks/h00s/kruno">>,
<<"{"_type":"location","acc":10,"alt":196,"batt":0,"lat":,"lon":,"tid":"kr","tst":1552035680,"vac":20,"vel":0}">>,
{1552,35700,
23052}},
[{route,
'emqx@',
<<"owntracks/h00s/kruno">>},
{dispatch,
<<"owntracks/+/+">>,
1},
{route,
'emqx@',
<<"owntracks/#">>}]}]},
undefined},
15000]}}}
erl_child_setup closed

Crash dump is being written to: log/crash.dump...done
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
[os_mon] memory supervisor port (memsup): Erlang has closed

@gilbertwong96 gilbertwong96 self-assigned this Mar 8, 2019
@gilbertwong96 gilbertwong96 added this to the 3.1-beta.2 milestone Mar 8, 2019
@karlem

This comment has been minimized.

Copy link

@karlem karlem commented Mar 11, 2019

I have to same issue on EMQ: v3.0.0

@turtleDeng turtleDeng added Enhancement and removed Support labels Mar 12, 2019
@turtleDeng turtleDeng assigned linjunjj and unassigned gilbertwong96 Mar 12, 2019
@linjunjj

This comment has been minimized.

Copy link
Contributor

@linjunjj linjunjj commented Mar 12, 2019

@h00s I can't reproduce this issues,Please provide the emqx.conf file and log file

@linjunjj

This comment has been minimized.

Copy link
Contributor

@linjunjj linjunjj commented Mar 15, 2019

@h00s The problem has been fixed and will be released in beta.2 :)

@h00s

This comment has been minimized.

Copy link
Author

@h00s h00s commented Mar 15, 2019

@h00s The problem has been fixed and will be released in beta.2 :)

Sorry for not providing emqx.conf file, I've been quite busy last few days.
But, my emqx.conf is basic, only changes on cluster cookie, node name and anonymous set to false.

Good to hear it will be fixed! :) Thanks for the fix and all the hard work, will test it when beta.2 is released!

@gilbertwong96

This comment has been minimized.

Copy link
Contributor

@gilbertwong96 gilbertwong96 commented Mar 18, 2019

Hi, @h00s , it has been fixed in emqx3.1 beta2 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.