New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connection errors when sending large amounts of data to browser via a bokeh server #7374

Closed
Laurens-T opened this Issue Jan 5, 2018 · 17 comments

Comments

Projects
None yet
7 participants
@Laurens-T
Copy link

Laurens-T commented Jan 5, 2018

Recently I came across a problem with sending lots of data to a bokeh server, between 20 - 50 MB. The command I used was

bokeh serve data_visualisation --show --args -i path_to_large_file

What would happen is that my plot would render, and then almost instantly a timeout would occur so no communication between the browser and server was possible:

2018-01-05 10:19:58,280 200 GET /data_visualisation (::1) 9227.38ms
2018-01-05 10:19:58,455 101 GET /data_visualisation/ws?bokeh-protocol-version=1.0&bokeh-session-id=jbQmzoBZbYDBDUdi5Ix6QShbAJtAMGPhmKvC1SWjOuwA (::1) 0.87ms
2018-01-05 10:19:58,455 WebSocket connection opened
2018-01-05 10:19:58,456 ServerConnection created
2018-01-05 10:21:28,512 WebSocket connection closed: code=None, reason=None

In the browser console, the following errors where shown:

[bokeh] Lost websocket 0 connection, 1006 ()  connection.js:221
[bokeh] Websocket connection 0 disconnected, will not attempt to reconnect connection.js:105 

The problem was solved by editing tornado.py by changing the rule on line 221
super(BokehTornado, self).__init__(all_patterns)
to
super(BokehTornado, self).__init__(all_patterns, websocket_max_message_size=50 * 1024 * 1024)

By default, the maximum socket message size is 10MiB for the used version of Tornado, and would disconnect for this reason. A more meaningful error message or the possibility to change this value in bokeh would be appreciated.

Software version info

bokeh==0.12.13
Tornado==4.5.2
python==3.6.3
MacOS High Sierra (10.13.2)
Any browser (Chrome (63.0.3239.84) / Safari / Firefox)

@bryevdv

This comment has been minimized.

Copy link
Member

bryevdv commented Jan 5, 2018

@Belloco unfortunately it is not possible to provide a better error message. Those messages originate from the browser, and Tornado respectively, not Bokeh. Bokeh only sees that the connection is closed, it does not get any information about why it was closed.

However, if it is possible to expose websocket_max_message_size as a user configurable option that that's certainly reasonable. However it is not completely trivial, as that option appears to be new to Tornado 4.5 and Bokeh support tornado >=4.4 so there will have to be some kind of version handling. Are you interested in helping work up a PR for this?

@bryevdv bryevdv added this to the 0.12.14 milestone Jan 5, 2018

@bryevdv bryevdv modified the milestones: 0.12.14, 0.12.x Jan 5, 2018

@Laurens-T

This comment has been minimized.

Copy link
Author

Laurens-T commented Jan 6, 2018

@bryevdv I'm not really interested in helping work up a PR. I mainly wanted to make sure that if other people encountered this problem, they wouldn't have to search as long as I did :)

@bryevdv

This comment has been minimized.

Copy link
Member

bryevdv commented Jan 9, 2018

@Belloco FWIW open source software depends crucially on spreading the workload to more people. Since you seem capable of doing this work, I do hope you might reconsider. Thanks for the detailed report in any case.

@jnettels

This comment has been minimized.

Copy link
Contributor

jnettels commented Jan 17, 2018

THANK YOU so much for posting this solution! This was a huge problem for me, but by chance I stumbled across this post.

@timothydmorton

This comment has been minimized.

Copy link
Contributor

timothydmorton commented Jan 17, 2018

I'll second the thanks for this solution, which has allowed me a temporary workaround to this problem. If anyone is able to work up a PR to help fix this as a configurable option in a future version that would be awesome.

@jnettels

This comment has been minimized.

Copy link
Contributor

jnettels commented Apr 3, 2018

After updating to Bokeh 0.12.15 I wanted to apply the "fix" to the file tornado.py and discovered that the relevant line is now 263 and has been changed to (in #7680):

super(BokehTornado, self).__init__(all_patterns, **kwargs)

Can **kwargs somehow be used to set websocket_max_message_size in a "regular" Bokeh server application?
Or would that require an "embedded" approach (something I have not tried yet)?

@kkgadiraju

This comment has been minimized.

Copy link

kkgadiraju commented Jun 4, 2018

Has anyone found a solution to this issue yet? Or is switching to datashader the only solution for now?

@jbednar

This comment has been minimized.

Copy link
Contributor

jbednar commented Jun 4, 2018

Seems like switching to datashader is only one solution, and that the more direct solution is to change the tornado setting as outlined above.

@bryevdv

This comment has been minimized.

Copy link
Member

bryevdv commented Jun 4, 2018

I'm working on a PR for this right now.

@bryevdv bryevdv referenced this issue Jun 4, 2018

Merged

add option to set websocket_max_message_size #7955

3 of 3 tasks complete

@bryevdv bryevdv modified the milestones: 0.13.x, 0.13.0 Jun 4, 2018

@bryevdv bryevdv closed this in #7955 Jun 5, 2018

@FunnyGoofy

This comment has been minimized.

Copy link

FunnyGoofy commented Jul 10, 2018

is this problem solved? thanks

@bryevdv

This comment has been minimized.

Copy link
Member

bryevdv commented Jul 10, 2018

--websocket-max-message-size is available in 0.13.0

@FunnyGoofy

This comment has been minimized.

Copy link

FunnyGoofy commented Jul 10, 2018

thanks a lot, after updated to 0.13.0, the error is gone

@FunnyGoofy

This comment has been minimized.

Copy link

FunnyGoofy commented Jul 10, 2018

if I try to select item from combox and call on_change(). console says: WARNING: bokeh.server.views.ws:Failed sending message as connection was closed

@bryevdv

This comment has been minimized.

Copy link
Member

bryevdv commented Jul 10, 2018

--websocket-max-message-size is an option you have to set yourself. Tornado recently introduced a max default web socket message size. All we can do is expose the parameter to override it, which we have done.

@FunnyGoofy

This comment has been minimized.

Copy link

FunnyGoofy commented Jul 11, 2018

I overwrote it like:
super(BokehTornado, self).__init__(all_patterns, websocket_max_message_size=50 * 1024 * 1024, **kwargs)

but, when I switch item from combox and update plot, I got error like this one.

ERROR:bokeh.server.protocol_handler:error handling message Message 'PATCH-DOC' (revision 1): KeyError('JobNumber',)

Am I not sure if my problem is associated with https://github.com/bokeh/bokeh/issues/4000

Thanks a lot

@bryevdv

This comment has been minimized.

Copy link
Member

bryevdv commented Jul 11, 2018

@FunnyGoofy your last messages have nothing do do with this issue. I am deleting your them because it is unhelpful and confusing to have unrelated messages mixed in together. Please start with the mailing list to see if there needs to be a new issue made.

@bokeh bokeh deleted a comment from FunnyGoofy Jul 11, 2018

@bokeh bokeh deleted a comment from FunnyGoofy Jul 11, 2018

@bokeh bokeh deleted a comment from FunnyGoofy Jul 11, 2018

@bryevdv

This comment has been minimized.

Copy link
Member

bryevdv commented Jul 11, 2018

@FunnyGoofy 50 is an incredibly tiny value. The units of websocket_max_message_size are bytes (as documented) so setting to 50 means that any message larger then 50 bytes will cause the connection to be terminated. You want to set to a very large value.

@bokeh bokeh deleted a comment from FunnyGoofy Jul 11, 2018

@bokeh bokeh locked as off topic and limited conversation to collaborators Jul 11, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.