Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check inactive WebSocket/SockJS sessions before they're connected on the STOMP level [SPR-11884] #16503

Closed
spring-issuemaster opened this issue Jun 18, 2014 · 2 comments

Comments

Projects
None yet
2 participants
@spring-issuemaster
Copy link
Collaborator

commented Jun 18, 2014

Prashant Deva opened SPR-11884 and commented

This is referencing the issue I wsa discussing with Rossen over email during the begining of the week.

Its reproducible on both Tomcat 7.0.53 and 8.0.8.

The number of files created by Spring keeps on increasing if there are a high number of connections (100s or 1000s). Increasing limit using ulimit is of no use since the number of files created will keep growing till the limit is hit, no matter what limit is set.

The issue is not visible when there are only a small number of connections.

I dont know of a good way to simulate hundreds or thousands of users so I have given Rossen full access to our production servers (over email) to reproduce the issue, including the source code of our app.

This is a blocker since our app crashes within minutes of starting and there is nothing we can do to keep it up


Affects: 4.0.5

Referenced from: commits 2b1ff4c, 40c203c, d18fc53, 113fd11, 7dc2b29, 86de416, 618771d, 3af488a, a3fa9c9

@spring-issuemaster

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 18, 2014

Rossen Stoyanchev commented

From initial investigation it appears the count of connections on the Tomcat side is about 10x the count of connections to the STOMP broker relay side (they should be about the same) as shown through lsof -p <pid> -i -n -P -a and then grepping for the STOMP broker TCP port (61613 by default) orand the Tomcat server id and port (e.g. 127.0.0.1:80).

Early indications point to an issue on the Tomcat side but it's not clear what the cause is yet. The access logs show just under 2 million websocket requests made with the vast majority using the WebSocket transport and much smaller numbers (50-100K) of xhr_streaming and xhr requests.

@spring-issuemaster

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 29, 2014

Rossen Stoyanchev commented

A more detailed investigation showed WebSocket/SockJS sessions that manage to get established successfully at first but never go as far as connecting on the STOMP level. This is often an indication of proxy issues but can also be a slow network.

For client sessions that do connect on the STOMP level, when a heartbeat interval is negotiated on startup, the STOMP broker automatically closes connections after a period of inactivity (or slow activity). However for sessions that never connect on the STOMP level there is no such protection.

To deal with this, a change has been introduced to check for WebSocket/SockJS sessions that have never received any messages within 60 seconds of connecting and close them. This is effectively makes up for the lack of heartbeats before the session connects on the level of STOMP.

Thanks Prashant Deva for letting me work with you on this and test it out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.