Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connected clients slow down dramatically when server port is connected without receiving anything #13298

Closed
wangxhere opened this issue Jun 14, 2018 · 5 comments

Comments

@wangxhere
Copy link

@wangxhere wangxhere commented Jun 14, 2018

Pre-Condition

HC cluster with 2 nodes, running as embedded servers. There is a health monitoring script on each of the nodes which connect to the server port 5701 using nc every half minute, disconnects after 1 seconds without sending anything.

There are 2 clients connecting to the cluster using Java client. A 1-second timeout is set for all the operations performed by those clients.

Scenario

There is a high chance client-side operation would slow down around the time when nc on server instance tries to connect and disconnect, causing client procedure to timeout.

This scenario is replicable in different network and server setups on our side.

Expected Result

As the health check is not even trying to connect to server port in a malicious way, it is expected that this should not affect normal client-side operation.

@mmedenjak mmedenjak added this to the 3.11 milestone Aug 27, 2018
@sancar
Copy link
Member

@sancar sancar commented Sep 5, 2018

Hi @wangxhere ,
Can you post the exact command that you use including the parameters you pass with nc ?
A 1-second timeout is set for all the operations performed by those clients.
Which property exactly you are referring to ? If there is any other special config you use, can you share them with us also ?

@sancar sancar added the Type: Defect label Sep 13, 2018
@tezc
Copy link
Contributor

@tezc tezc commented Sep 14, 2018

Hi @wangxhere ,
We are trying to reproduce the issue. In addition to @sancar 's, can you share some more details with us? What is the version of Hazelcast? Do clients reconnect each time? (Client connect, perform single op, disconnect vs Client connect once, perform all operations).

@wangxhere
Copy link
Author

@wangxhere wangxhere commented Sep 14, 2018

@tezc @sancar
I was using nc -w 1 127.0.0.1 5701 to connect to the port and immediately disconnect (without sending anything). During doing this, the other clients were experiencing pauses. Version of Hazelcast was 3.9.2 (embedded in own app). Normal client does not reconnect but maintains long connection at all times. Only that nc disconnects each time.

@tezc
Copy link
Contributor

@tezc tezc commented Sep 14, 2018

Hi @wangxhere ,
This issue is fixed in 3.10.3, see #13363
I can only reproduce it with 3.10.2 and below. Please upgrade and test it with 3.10.3 and above.

@tezc
Copy link
Contributor

@tezc tezc commented Sep 17, 2018

Closing the issue, please open a new issue if you see this problem in newer versions.

@tezc tezc closed this Sep 17, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.