TypeError: buf.copy is not a function

[noamshemesh]

Hi,
we use primus 4.0.4 with engine.io 1.6.4 and node 4.2.4.
to stream data from db to client we use spark.write().
it works well for almost all the times but when there is a lot of data to stream it's crashing with this error:

TypeError: buf.copy is not a function
    at Function.Buffer.concat (buffer.js:238:9)
    at IncomingMessage.onData (/opt/bigpanda/web-api/node_modules/engine.io/lib/transports/polling.js:160:23)
    at emitOne (events.js:77:13)
    at IncomingMessage.emit (events.js:169:7)
...(wrappers)...
    at IncomingMessage.Readable.read (_stream_readable.js:360:10)
    at flow (_stream_readable.js:743:26)
    at resume_ (_stream_readable.js:723:3)

after enabling DEBUG for engine.io pulling, we see these log lines:

engine:polling setting request +25s
engine:polling writing "97:0{"sid":"<id>"
,"upgrades":["websocket"],"pingInterval":25000,"pingTimeout
":60000}" +0ms
engine:polling setting request +5s 
engine:polling received "21:4{"type":"subscribe"}" +183ms
engine:polling writing "1:6" +167ms

I'm pretty sure that it's an engine.io problem, but maybe I wrong.

[3rd-Eden]

Yes, it is indeed an Engine.IO issue and it's something that we cannot fix from Primus as it's the part where it receives messages from the browser to the server. So it's not the spark.write method that is the issue here but engine.io actually receiving data. The only thing we could do to fix it is to patch the prototype of that method or start forking.

[noamshemesh]

ok, thanks. I let them know socketio/engine.io#356

[lpinca]

@noamshemesh try to monkey patch engine.io and see what happens here https://github.com/socketio/engine.io/blob/a31551e141d3ea227e3f85b93285007ca16e4de0/lib/transports/polling.js#L160

Probably chunks is a string and data is a buffer, but how is that even possible?

[noamshemesh]

@lpinca thanks, that what we did. we saw that the client sends one big object to the client over pulling and then the server crashes.
as a temporary fix, we divided the payload to smaller payloads, that fixed the problem for now.

[lpinca]

I tried reproducing this streaming a big file (~300MB) from client to server but it works fine.
Maybe it's because I'm using the node client or the setup is somehow different.

[noamshemesh]

did you set allowUpgrades for engine.io to false?

our setup for primus client is:

     { 
        reconnect: {
          maxDelay: 60000,
          minDelay: 500,
          retries: 100
        },
        ping: 60000,
        pong: 15000,
        timeout: 40000
      }

[lpinca]

I used the websockets option in the client and set it to false.

[lpinca]

This is the code I'm using:

'use strict';

const Primus = require('primus');
const http = require('http');
const fs = require('fs');

const server = http.createServer();
const primus = new Primus(server, { transformer: 'engine.io' });

primus.on('connection', spark => {
  console.log(spark.id, 'connected');

  spark.on('end', () => primus.destroy());
  spark.pipe(fs.createWriteStream(__dirname + '/copy.txt'));
});

primus.on('disconnection',  spark => console.log(spark.id, 'disconnected'));

server.listen(3000, () => {
  const socket =  new primus.Socket('http://localhost:3000', {
    websockets: false
  });

  fs.createReadStream(__dirname + '/300MB.txt', { encoding: 'utf8' })
    .on('data', data => socket.write(data))
    .on('end', () => socket.end());
});

[noamshemesh]

seems like a slim version of what we do. we even write less data than 300mb.
I'm closing this. if someone else encounters a similiar problem, please reference to here

[lpinca]

I wonder if it's a middleware or a plugin that causes the issue.

[noamshemesh]

we don't use any middleware for primus itself. we use only authorize to validate the connection.
also I'm not sure it's something between as it fails on IncomingMessage.onData which is pretty the entry point when a message arrives.
does it rule this suspect out?

[lpinca]

The same issue was happening when using the authorization middleware. I'll try adding that.

[noamshemesh]

interesting! waiting for updates

[lpinca]

Still no luck, I've added

primus.authorize((req, next) => setTimeout(next, 50));

to the previous code, but it works.