Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broker started dropping WebSocket connections #2619

Open
mdrobny opened this issue Jun 13, 2019 · 4 comments
Assignees
Labels

Comments

@mdrobny
Copy link
Contributor

@mdrobny mdrobny commented Jun 13, 2019

Environment

Description

Hi, me again :)

My EMQX cluster was up for ~12 days without any problems but suddenly today it started to drop connected clients and it wasn't accepting new ones. I am almost sure all those clients were connecting via WebSockets (/mqtt).

I see those errors in the log/emqx.log.1 file:

2019-06-13 07:55:29.233 [error] Ranch listener 'mqtt:ws' had connection process started with cowboy_clear:start_link/4 at <0.20496.1909> exit with reason: {function_clause,[{emqx_ws_connection,websocket_handle,[{text,<<16>>},{state,undefined,undefined,{{172,20,0,4},60812},{{172,20,55,40},8083},15000,{pstate,external,#Fun<emqx_ws_connection.2.116361389>,{{172,20,0,4},60812},undefined,4,<<"MQTT">>,<<>>,false,<0.20496.1909>,undefined,undefined,undefined,undefined,false,#{},1048576,undefined,undefined,undefined,false,true,true,false,disconnect,#{msg => 0,pkt => 0},#{msg => 0,pkt => 0},false,undefined,false,#{from_client => 0,to_client => 0},emqx_ws_connection,#{},[]},{none,#{max_packet_size => 1048576,version => 4}},undefined,true,undefined,undefined}],[{file,"src/emqx_ws_connection.erl"},{line,183}]},{cowboy_websocket,handler_call,6,[{file,"src/cowboy_websocket.erl"},{line,471}]},{cowboy_http,loop,2,[{file,"src/cowboy_http.erl"},{line,233}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2019-06-13 09:11:59.001 [error] 172.20.3.137:17508 crasher:
    initial call: cowboy_clear:connection_process/4
    pid: <0.15125.1924>
    registered_name: []
    exception error: no function clause matching
                     emqx_ws_connection:websocket_handle({text,<<16>>},
                                                         {state,undefined,
                                                          undefined,
                                                          {{172,20,3,137},
                                                           17508},
                                                          {{172,20,55,40},
                                                           8083},
                                                          15000,
                                                          {pstate,external,
                                                           #Fun<emqx_ws_connection.2.116361389>,
                                                           {{172,20,3,137},
                                                            17508},
                                                           undefined,4,
                                                           <<"MQTT">>,<<>>,
                                                           false,
                                                           <0.15125.1924>,
                                                           undefined,
                                                           undefined,
                                                           undefined,
                                                           undefined,false,
                                                           #{},1048576,
                                                           undefined,
                                                           undefined,
                                                           undefined,false,
                                                           true,true,false,
                                                           disconnect,
                                                           #{msg => 0,
                                                             pkt => 0},
                                                           #{msg => 0,
                                                             pkt => 0},
                                                           false,undefined,
                                                           false,
                                                           #{from_client => 0,
                                                             to_client => 0},
                                                           emqx_ws_connection,
                                                           #{},[]},
                                                          {none,
                                                           #{max_packet_size =>
                                                              1048576,
                                                             version => 4}},
                                                          undefined,true,
                                                          undefined,
                                                          undefined}) (src/emqx_ws_connection.erl, line 183)
      in function  cowboy_websocket:handler_call/6 (src/cowboy_websocket.erl, line 471)
      in call from cowboy_http:loop/2 (src/cowboy_http.erl, line 233)
    ancestors: [<0.2201.0>,<0.2200.0>,ranch_sup,<0.1969.0>]
    message_queue_len: 0
    messages: []
    links: [#Port<0.11894492>,<0.2201.0>]
    dictionary: [{'$logger_metadata$',#{peername => "172.20.3.137:17508"}}]
    trap_exit: true
    status: running
    heap_size: 2586
    stack_size: 27
    reductions: 2480
  neighbours:

At the same time clients were connecting normally via MQTT protocol (not using WebSockets).

What can I do to fix it and prevent that in the future? Did you experience such behaviour, maybe it's fixed in newer versions?

I didn't find any issue with similar error

@gilbertwong96 gilbertwong96 self-assigned this Jun 14, 2019
@tradingtrace

This comment has been minimized.

Copy link
Contributor

@tradingtrace tradingtrace commented Jun 16, 2019

Hi @mdrobny
According to the error log above, could it be that the clients have sent a text frame through WebSocket to the server?

In MQTT specification:

MQTT Control Packets MUST be sent in WebSocket binary data frames. If any other type of data frame is received the recipient MUST close the Network Connection [MQTT-6.0.0-1].

@Gilbert-Wong Should a function clause be added here to match the text frame and close the WebSocket connection?

%% Pings should be replied with pongs, cowboy does it automatically
%% Pongs can be safely ignored. Clause here simply prevents crash.
websocket_handle(Frame, State)
when Frame =:= ping; Frame =:= pong ->
{ok, ensure_stats_timer(State)};
websocket_handle({FrameType, _}, State)
when FrameType =:= ping; FrameType =:= pong ->
{ok, ensure_stats_timer(State)}.

@gilbertwong96

This comment has been minimized.

Copy link
Contributor

@gilbertwong96 gilbertwong96 commented Jun 20, 2019

Yes, of course. @tradingtrace, thanks for your advice.

@gilbertwong96 gilbertwong96 added the MQTT5 label Jun 20, 2019
@mdrobny

This comment has been minimized.

Copy link
Contributor Author

@mdrobny mdrobny commented Jun 21, 2019

@tradingtrace thanks for the response.

I think clients were sending WebSocket binary data frames because they all use mqtt.js library which handles WebSocket connections properly.
Additionally I think those clients were connected already and they started to be disconnected so I don't think way of changing data could change :/

@gilbertwong96

This comment has been minimized.

Copy link
Contributor

@gilbertwong96 gilbertwong96 commented Jul 3, 2019

@mdrobny, according to the error log you attach, these clients have been connected already. However, when you send websocket message whose frame type is text, they will be disconnected.

@turtleDeng turtleDeng assigned tigercl and unassigned gilbertwong96 Sep 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.