Performance of ss.createBlobReadStream(blob).pipe(stream)

[binarykitchen]

Hello there

I experience performance issues with the following piece of code:

            var stream      = ss.createStream(),
                self        = this,
                framesCount = self.framesCount;

            this.canvas.toBlob(function(blob) {

                // this is too slow!

                ss(self.socket).emit(
                    'newFrame',
                    stream,
                    {
                        framesCount: framesCount
                    }
                );

                ss.createBlobReadStream(blob).pipe(stream);

            }, 'image/jpeg', this.settings.video.quality);

My goal is to send images to the server in binary. I made measurements and found out that ss.createBlobReadStream(blob).pipe(stream); is the slowest part.

Firefox alone needs about 50ms - 60ms to pipe that blob to the server. In other words the main thread is blocked by 50ms - 60ms!

Any clues?

[nkzawa]

Thank you for reporting it.
Could you tell me the details of how you benchmarked?

I want to know which part is the problem, since many things will happen asynchronously when you execute ss.createBlobReadStream(blob).pipe(stream);.

[binarykitchen]

Benchmark with Date.now() and console.log.

var stream      = ss.createStream(),
    self        = this,
    framesCount = self.framesCount;

var start1 = Date.now();

this.canvas.toBlob(function(blob) {

    ss(self.socket).emit(
        'newFrame',
        stream,
        {
            framesCount: framesCount
        }
    );

    var blobStream = ss.createBlobReadStream(blob);

    blobStream.on('end', function() {
        var end = Date.now();

        console.log('Time1 (ms)', end - start1);
        console.log('Time2 (ms)', end - start2);
    });

    var start2 = Date.now();
    blobStream.pipe(stream);

}, 'image/jpeg', this.settings.video.quality);

Will output something like:

Time1 (ms) 160
Time2 (ms) 124

You can see that Time2 measures the time for piping alone which consumes over 70% of the whole code snippet which is not good.

[nkzawa]

Ah, got it.

Sending data is almost finished when end event happens, which means Time2 includes time of network communications.

And data is sent gradually and asynchronously, so the main thread is not blocked.

If you need more performance, highWaterMark option may help.

ss.createStream({hightWaterMark: 16 * 1024});
ss.createBlobReadStream(blob, {hightWaterMark: 16 * 1024});

[binarykitchen]

@nkzawa Thanks! Can you explain what highWaterMark means? Where it is documented?

[binarykitchen]

@nkzawa And I do not want a writable stream when sending the image to the server. Can I get a readable stream only? (I suspect this because hightWaterMark is for writable streams only)

[nkzawa]

highWaterMark is both for readable and writable stream.

see the document: http://nodejs.org/api/stream.html

[binarykitchen]

I see. Have implemented something like this to automatically adjust the buffer size:

            this.canvas.toBlob(function(blob) {

                ss(self.socket).emit(
                    'newFrame',
                    stream,
                    {
                        framesCount: framesCount
                    }
                );

                var blobStream = ss.createBlobReadStream(blob, {hightWaterMark: self.hightWaterMark}),
                    size = 0;

                blobStream.on('data', function(chunk) {
                    size += chunk.length;
                });

                blobStream.on('end', function() {
                    // automatically correct max buffer size for the next image frame
                    self.hightWaterMark = size * 1.1;
                });

                blobStream.pipe(stream);

            }, 'image/jpeg', this.settings.video.quality);

[nkzawa]

I'm not sure whether setting hightWaterMark higher than a blob size is good or not. You should try.

I will close the issue anyway.

[binarykitchen]

@nkzawa Thanks but sorry, I have quit this module and am using the websocket-stream one. It's pure binary and faster.