New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Hybi10 binaryType #511

Closed
ebidel opened this Issue Sep 7, 2011 · 30 comments

Comments

Projects
None yet
@ebidel
Copy link

ebidel commented Sep 7, 2011

Just landed in Chromium: http://code.google.com/p/chromium/issues/detail?id=93652

Allows sending Blob or ArrayBuffer (binary data).

@einaros

This comment has been minimized.

Copy link
Contributor

einaros commented Sep 8, 2011

The reason why I didn't implement this part of the hybi draft is simply that the other transports would struggle with it. It'll be better to add this when the draft stabilizes, and more browsers support it.

@ebidel

This comment has been minimized.

Copy link
Author

ebidel commented Sep 9, 2011

By other transports do you mean other protocol versions? Regardless of whether or not the current spec is stable, expect binary support to stay.

@ebidel

This comment has been minimized.

Copy link
Author

ebidel commented Oct 11, 2011

Is this fixed with v0.85? I see binary support in the release notes.

@einaros

This comment has been minimized.

Copy link
Contributor

einaros commented Oct 11, 2011

Well it's in the parser at least, but it has yet to be connected to anything meaningful.

Now is a good time to ask: What binary data do you actually want to submit?

@ebidel

This comment has been minimized.

Copy link
Author

ebidel commented Oct 11, 2011

Use cases are many. Here are 2 off the top of my head:

  • stream audio to connected clients
  • realtime screen collaboration tool: stream the binary images across the wire, render to canvas.

https://github.com/Worlize/WebSocket-Node seems to support binary transfer.

@ebidel ebidel closed this Oct 11, 2011

@ebidel ebidel reopened this Oct 11, 2011

@einaros

This comment has been minimized.

Copy link
Contributor

einaros commented Oct 11, 2011

Supporting binary transfer is, as I said, no big deal. I just struggle to see real use cases, seeing as js on the client has no binary data type.

Stream audio: How would you decode the binary efficiently, and assign them to the float arrays used by the sound apis (which are still not standard)?

Stream binary images: Canvas painting would either (a) be a video element which is drawn to a canvas, (b) an image which is drawn to a canvas, (c) rgba number values set to the bitmapdata of the canvas. Of the three, neither could really use binary data sent via websockets. And (c), being the closest to even remotely possible, is insanely slow.

Edit: I see that Chrome at least support receiving arraybuffers now, which was ironically your opening statement for this issue, so I should have caught that. Anyhoo, the audio bit should be doable as you say, and probably canvas as well if it now has a setter for the typed arrays.

@ebidel

This comment has been minimized.

Copy link
Author

ebidel commented Oct 11, 2011

js on the client has no binary data type.

Yep. They're many new types in JS to work with binary data: Typed Arrays, ArrayBuffer, Blob, File.
They're used all over the place now (webgl, audio apis, file apis, xhr2). I presented an entire section on this at Google IO:
http://www.htmlfivewow.com/slide11

Decoding audio:

The Web Audio API for example (now supported in Chrome and WebKit nightlies) has methods for decoding audio:
https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/specification.html#AudioContext-section

You send it an ArrayBuffer.

streaming binary images

I was thinking more like a remote desktop client. Images are rapidly sent over a websocket (without the need to base64 encode, etc.) and rendered onto a canvas. c.) could also work, since the updated protocol allows passing ArrayBuffers.

I believe they're swamping out canvas's CanvasPixelArray to use ArrayBuffer instead. Regardless, you'll
be able to .getBlob() from the canvas and create an image file from it:
http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#dom-canvas-toblob

@einaros

This comment has been minimized.

Copy link
Contributor

einaros commented Oct 11, 2011

I was familiar with the floating point types, but not blob / file and alike. Granted those, and also canvas' push for ArrayBuffer, it makes sense to complete this implementation sooner rather than later.

I'll see if I can find time to do so later this week.

@ebidel

This comment has been minimized.

Copy link
Author

ebidel commented Oct 11, 2011

You da man! Thank you!

@mmarcon

This comment has been minimized.

Copy link

mmarcon commented Jan 30, 2012

Was this ever implemented? I am curious to play with images on canvas to be sent to the server for heavy processing and ArrayBuffer over socket.io seems like a pretty solution.

@einaros

This comment has been minimized.

Copy link
Contributor

einaros commented Jan 30, 2012

I implemented this partially in socket.io, but fully in 'ws', my up-to-date websocket client/server for node. See https://github.com/einaros/ws

@ebidel

This comment has been minimized.

Copy link
Author

ebidel commented Feb 10, 2012

What is missing on the socket.io side of things?

@einaros

This comment has been minimized.

Copy link
Contributor

einaros commented Feb 10, 2012

  • Support across the other transports.
  • Merge of my pullreq which uses ws over my old parsers in socket.io, OR:
  • Switch to engine.io+websocket.io and exposure of ws' binary features through websocket.io.

I added more binary samples to ws, by the way.

@ebidel

This comment has been minimized.

Copy link
Author

ebidel commented Feb 10, 2012

einarios, this would be killer for socket.io to support. It's by far the most adopted
websocket implementation for node. Unfortunately, I'm seeing a lot of people starting
to abuse dataURLs (33% encoding overhead) for sending files. If socket.io supported
binary out of the box, we could speed up the web! :)

Please let me know how I can help.

@einaros

This comment has been minimized.

Copy link
Contributor

einaros commented Feb 10, 2012

Well in terms of websockets, socket.io will just be a front for ws, which already supports binary. The challenge at this point is the other transports. Someone will have to decide to deal with those.

@ebidel

This comment has been minimized.

Copy link
Author

ebidel commented Feb 10, 2012

Understood. We're in a good place with XHR2 supporting binary. I'm sure flash has some special sauce to plug into, but I can't speak for the flashsocket transport.

@einaros

This comment has been minimized.

Copy link
Contributor

einaros commented Feb 10, 2012

Flash itself should be able to do it, but it's running against websocket hixie, so it'd either require a hybi rewrite (quite a bit of work), or have binary data base64'd (ouch). In terms of the js / flash bridge, I don't know how the different flash versions would be able to deal with typed arrays and blobs. XHR2 may be the only reasonable option for non hybi browsers.

In either case, this will be after the engine.io switch. And someone will still have to step up to the task for xhr ;)

@ebidel

This comment has been minimized.

Copy link
Author

ebidel commented Feb 10, 2012

Feel free to ping me when that happens. Glad to hack something up.

@einaros

This comment has been minimized.

Copy link
Contributor

einaros commented Feb 11, 2012

It's up to @guille and @3rd-Eden. I keep track of websockets.

@briprowe

This comment has been minimized.

Copy link

briprowe commented Mar 13, 2012

I just wanted to add a vote for this feature. It would be very useful for the things I'm working on.

@adambratt

This comment has been minimized.

Copy link

adambratt commented Dec 4, 2012

Is there any chance of this being added? Would really be useful.

@ekryski

This comment has been minimized.

Copy link

ekryski commented Dec 22, 2012

+1 Would be great. Willing to hack on it for sure.

@itsatony

This comment has been minimized.

Copy link

itsatony commented Jan 10, 2013

+1

@rmzi

This comment has been minimized.

Copy link

rmzi commented Jan 14, 2013

https://github.com/liamks/Delivery.js -- Experimental Solution

@vote539

This comment has been minimized.

Copy link

vote539 commented May 24, 2013

So, Firefox and Chromium and even IE 10 now support the WebSocket.binaryType property.

If you turn on the experimental "serialization" option in my implementation of file uploads over Socket.IO, it is worth noting that in exchange for a 10-fold transmission overhead (my 1.9 MB binary file serialized to 25.2 MB of data), Socket.IO already provides a near-seamless transition between W3C's ArrayBuffer and Node's Buffer by way of serialization.

This seems like a good time to support WebSocket binary transmissions in Socket.IO; in cases that it's not supported, it is simple enough to internally resort to a base64 approach.

@Zolmeister

This comment has been minimized.

Copy link

Zolmeister commented Sep 8, 2013

+1

@rauchg

This comment has been minimized.

Copy link
Contributor

rauchg commented Sep 9, 2013

@vote539 I agree with this completely. I'll focus on binary after I get 1.0 out

@Rob-pw

This comment has been minimized.

Copy link

Rob-pw commented Jan 15, 2014

@guille Could we have a sitrep on this feature?

@mekwall

This comment has been minimized.

Copy link

mekwall commented Jan 24, 2014

I am also wondering what's going on with this. I am looking to use it to stream audio and video.

@vongosling

This comment has been minimized.

Copy link

vongosling commented Mar 10, 2014

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment