Skip to content

Keepalive ping() possibly causing connection to die #922

@attzonko

Description

@attzonko

Hello, I think we are seeing an issue where the ping frame trying to keep the connection alive, might be possibly causing the connection to get killed. Here is what I see once I enable DEBUG. Notice the data being sent with the ping frame.

client > Frame(fin=True, opcode=9, data=b'\x03\xbd\xdag', rsv1=False, rsv2=False, rsv3=False)
DEBUG:websockets.protocol:client > Frame(fin=True, opcode=9, data=b'\x03\xbd\xdag', rsv1=False, rsv2=False, rsv3=False)
client ! timed out waiting for pong
DEBUG:websockets.protocol:client ! timed out waiting for pong
client ! failing OPEN WebSocket connection with code 1011
DEBUG:websockets.protocol:client ! failing OPEN WebSocket connection with code 1011
client - state = CLOSING
DEBUG:websockets.protocol:client - state = CLOSING
client > Frame(fin=True, opcode=8, data=b'\x03\xf3', rsv1=False, rsv2=False, rsv3=False)
DEBUG:websockets.protocol:client > Frame(fin=True, opcode=8, data=b'\x03\xf3', rsv1=False, rsv2=False, rsv3=False)
client ! timed out waiting for TCP close
DEBUG:websockets.protocol:client ! timed out waiting for TCP close
client x closing TCP connection
DEBUG:websockets.protocol:client x closing TCP connection

On the server-side I see the following in the logs:

{"level":"debug","ts":1616198276.1780987,"caller":"app/web_conn.go:190","msg":"Error while reading message from websocket","error":"error during decoding websocket message: invalid character '\\x03' looking for beginning of value","errorVerbose":"invalid character '\\x03' looking for beginning of value\nerror during decoding websocket message\ngithub.com/mattermost/mattermost-server/v5/app.(*WebConn).ReadMsg\n\tgithub.com/mattermost/mattermost-server/v5/app/web_conn.go:257\ngithub.com/mattermost/mattermost-server/v5/app.(*WebConn).startPoller.func1.1\n\tgithub.com/mattermost/mattermost-server/v5/app/web_conn.go:188\nruntime.goexit\n\truntime/asm_amd64.s:1374"}

The invalid character message seems to change depending on what the data in the ping frame from the client looks like.

If I disable the pings ping_interval=None issue goes away. We don't want to disable the pings or try and keep the connection alive on our own. But we don't know if there is an issue with the server-side code or client.

Thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions