Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some error occuered after update 3.0.0 #172

Closed
DingGGu opened this issue Jan 13, 2017 · 25 comments
Closed

Some error occuered after update 3.0.0 #172

DingGGu opened this issue Jan 13, 2017 · 25 comments

Comments

@DingGGu
Copy link
Contributor

DingGGu commented Jan 13, 2017

I update socket.io-redis 3.0.0 right now,
But few Unknown error occured from error logs.

msgpack-js:msgpack in Decoder.parse
error: Unknown type 0xc7
msgpack-lite.lib:flex-buffer in DecodeBuffer.reserve
error: BUFFER_SHORTAGE
msgpack-js:msgpack in Object.decode
error-686655034 trailing bytes

Maybe It caused by "msgpack-lite": "0.1.26"

@DingGGu DingGGu changed the title msgpack-js:msgpack in Decoder.parse errorUnknown type 0xc7 Some error occuered after update 3.0.0 Jan 13, 2017
@darrachequesne
Copy link
Member

Yes, we switched to msgpack-lite here, in order to fix other decoding issues...

Are you able to reproduce consistently?

@DingGGu
Copy link
Contributor Author

DingGGu commented Jan 13, 2017

Yes, It happened repeatedly and even now.

Error: BUFFER_SHORTAGE
 File "<PROJECT>/node_modules/msgpack-lite/lib/flex-buffer.js", line 64, col 41, in DecodeBuffer.reserve
   if (end > this.buffer.length) throw new Error(BUFFER_SHORTAGE);
 File "<PROJECT>/node_modules/msgpack-lite/lib/read-format.js", line 117, col 23, in uint8
   var start = decoder.reserve(1);
 File "<PROJECT>/node_modules/msgpack-lite/lib/read-token.js", line 152, col 15, in null.<anonymous>
   var len = lenFunc(decoder);
 File "<PROJECT>/node_modules/msgpack-lite/lib/read-core.js", line 25, col 12, in decode
   return func(decoder);
 File "<PROJECT>/node_modules/msgpack-lite/lib/read-format.js", line 52, col 12, in map_to_obj
   v[i] = decode(decoder);
 File "<PROJECT>/node_modules/msgpack-lite/lib/read-token.js", line 159, col 12, in null.<anonymous>
   return method(decoder, len);
 File "<PROJECT>/node_modules/msgpack-lite/lib/read-core.js", line 25, col 12, in decode
   return func(decoder);
 File "<PROJECT>/node_modules/msgpack-lite/lib/read-format.js", line 81, col 16, in array
   value[i] = decode(decoder);
 File "<PROJECT>/node_modules/msgpack-lite/lib/read-token.js", line 159, col 12, in null.<anonymous>
   return method(decoder, len);
 File "<PROJECT>/node_modules/msgpack-lite/lib/read-core.js", line 25, col 12, in Codec.decode
   return func(decoder);
 File "<PROJECT>/node_modules/msgpack-lite/lib/decode-buffer.js", line 26, col 21, in DecodeBuffer.fetch
   return this.codec.decode(this);
 File "<PROJECT>/node_modules/msgpack-lite/lib/flex-buffer.js", line 166, col 28, in DecodeBuffer.read
   if (!length) return this.fetch();
 File "<PROJECT>/node_modules/msgpack-lite/lib/decode.js", line 10, col 18, in Object.decode
   return decoder.read();
 File "<PROJECT>/node_modules/socket.io-redis/index.js", line 145, col 24, in Redis.onmessage
   var args = msgpack.decode(msg);
 File "events.js", line 106, col 13, in emitTwo
 File "events.js", line 191, col 7, in RedisClient.emit
 File "<PROJECT>/node_modules/socket.io-redis/node_modules/redis/index.js", line 776, col 18, in return_pub_sub
   self.emit('messageBuffer', reply[1], reply[2]);
 File "<PROJECT>/node_modules/socket.io-redis/node_modules/redis/index.js", line 825, col 9, in RedisClient.return_reply
   return_pub_sub(this, reply);
 File "<PROJECT>/node_modules/socket.io-redis/node_modules/redis/index.js", line 188, col 18, in JavascriptRedisParser.Parser.returnReply
   self.return_reply(data);
 File "<PROJECT>/node_modules/redis-parser/lib/parser.js", line 491, col 12, in JavascriptRedisParser.execute
   this.returnReply(response)
 File "<PROJECT>/node_modules/socket.io-redis/node_modules/redis/index.js", line 267, col 27, in Socket.<anonymous>
   self.reply_parser.execute(buffer_from_socket);
 File "events.js", line 96, col 13, in emitOne
 File "events.js", line 188, col 7, in Socket.emit
 File "_stream_readable.js", line 172, col 18, in readableAddChunk
 File "_stream_readable.js", line 130, col 10, in Socket.Readable.push
 File "net.js", line 542, col 20, in TCP.onread

@darrachequesne
Copy link
Member

Are you using some external emitter (like socket.io-emitter or socket.io-php-emitter), that could publish packets that msgpack-lite would be unable to decode?

Or with several nodes, some that are using msgpack-lite and others still msgpack-js?

Related: #169

@DingGGu
Copy link
Contributor Author

DingGGu commented Jan 13, 2017

Yes of course. I use socket.io-emitter and I just upgraded few hours ago both socket.io-emitter and socket.io-redis

npm ls msgpack-lite
└└──┬┬ socket.io-emitter@2.0.0
  └└──── msgpack-lite@0.1.26

and there is no usage msgpack-js

@darrachequesne
Copy link
Member

@DingGGu may I ask what your setup is?

  • Redis server version?
  • Redis client version (node_redis, ioredis)?

I'd like to make sure that this issue is really related to msgpack-lite, before maybe switching back to msgpack-js..

@mavrick
Copy link

mavrick commented Jan 15, 2017

I'm getting the same error with my setup:

Redis Server:

  redis_version: 2.8.12
  redis_git_sha1: 0
  redis_git_dirty: 0
  redis_mode: standalone
  os: Linux 3.2.0-48 virtual x86_64
  arch_bits: 64
  multiplexing_api: epoll
  gcc_version: 4.6.3
  process_id: 7677628
  run_id: 
  tcp_port: 0
  uptime_in_seconds: 60156093
  uptime_in_days: 696
  hz: 10
  lru_clock: 0

npm versions:

"redis": "~2.6.3",
"socket.io": "1.7.2",
"socket.io-emitter": "1.1.0",
"socket.io-redis": "3.0.0"

code example:

var redisSub = redis.createClient(redis.port, redis.host, { detect_buffers: true, auth_pass: redis.pass});

redisSub.on("message", function(channel, message)
{
	console.log(channel, message);
});

var redisPub = redis.createClient(redis.port, redis.host, {auth_pass: redis.pass});

redisPub.publish("discord", JSON.stringify({type: "status", data: {uuid: id, status: status}}));

var socketio = require("socket.io");
var io = socketio.listen(server.server);
var ioredis = require("socket.io-redis");

var adapter = ioredis({
	pubClient: redisPub, 
	subClient: redisSub
});

io.adapter( adapter );

data being received:

socket.io#/# : '��y650LF��type\u0002�data��discord::status�\u0000dN4IgLgngDgpiBcIDOYCGYCuSQBoQBN1UFQMMBLfBEARgDYBmAFgE4BWABgHYuAOOxlwY0ATB1zI0mbIgD2AM3kAbcgDs4AXw1A==�nsp�/��roomsĥflags�'

stack trace:

Sun, 15 Jan 2017 10:23:11 GMT uncaughtException: BUFFER_SHORTAGE
Error: BUFFER_SHORTAGE
    at DecodeBuffer.reserve (c:\app\node_modules\msgpack-lite\lib\flex-buffer.js:64:41)
    at bin_buffer (c:\app\node_modules\msgpack-lite\lib\read-format.js:93:23)
    at c:\app\node_modules\msgpack-lite\lib\read-token.js:153:12
    at decode (c:\app\node_modules\msgpack-lite\lib\read-core.js:25:12)
    at map_to_obj (c:\app\node_modules\msgpack-lite\lib\read-format.js:52:12)
    at c:\app\node_modules\msgpack-lite\lib\read-token.js:159:12
    at decode (c:\app\node_modules\msgpack-lite\lib\read-core.js:25:12)
    at array (c:\app\node_modules\msgpack-lite\lib\read-format.js:81:16)
    at c:\app\node_modules\msgpack-lite\lib\read-token.js:159:12
    at Codec.decode (c:\app\node_modules\msgpack-lite\lib\read-core.js:25:12)
    at DecodeBuffer.fetch (c:\app\node_modules\msgpack-lite\lib\decode-buffer.js:26:21)
    at DecodeBuffer.read (c:\app\node_modules\msgpack-lite\lib\flex-buffer.js:166:28)
    at Object.decode (c:\app\node_modules\msgpack-lite\lib\decode.js:10:18)
    at Redis.onmessage (c:\app\node_modules\socket.io-redis\index.js:145:24)
    at emitTwo (events.js:106:13)
    at RedisClient.emit (events.js:191:7)
Sun, 15 Jan 2017 10:23:11 GMT uncaughtException: BUFFER_SHORTAGE
Error: BUFFER_SHORTAGE
    at DecodeBuffer.reserve (c:\app\node_modules\msgpack-lite\lib\flex-buffer.js:64:41)
    at bin_buffer (c:\app\node_modules\msgpack-lite\lib\read-format.js:93:23)
    at c:\app\node_modules\msgpack-lite\lib\read-token.js:153:12
    at decode (c:\app\node_modules\msgpack-lite\lib\read-core.js:25:12)
    at map_to_obj (c:\app\node_modules\msgpack-lite\lib\read-format.js:52:12)
    at c:\app\node_modules\msgpack-lite\lib\read-token.js:159:12
    at decode (c:\app\node_modules\msgpack-lite\lib\read-core.js:25:12)
    at array (c:\app\node_modules\msgpack-lite\lib\read-format.js:81:16)
    at c:\app\node_modules\msgpack-lite\lib\read-token.js:159:12
    at Codec.decode (c:\app\node_modules\msgpack-lite\lib\read-core.js:25:12)
    at DecodeBuffer.fetch (c:\app\node_modules\msgpack-lite\lib\decode-buffer.js:26:21)
    at DecodeBuffer.read (c:\app\node_modules\msgpack-lite\lib\flex-buffer.js:166:28)
    at Object.decode (c:\app\node_modules\msgpack-lite\lib\decode.js:10:18)
    at Redis.onmessage (c:\app\node_modules\socket.io-redis\index.js:145:24)
    at emitTwo (events.js:106:13)
    at RedisClient.emit (events.js:191:7)

@darrachequesne
Copy link
Member

redisPub.publish("discord", JSON.stringify({type: "status", data: {uuid: id, status: status}}));

@mavrick JSON.stringify? Shouldn't it be msgpack.encode instead? Else msgpack.decode will obviously fail when trying to parse JSON content.

@DingGGu
Copy link
Contributor Author

DingGGu commented Jan 16, 2017

I just downgraded version,
and then resolved.

@DingGGu
Copy link
Contributor Author

DingGGu commented Jan 16, 2017

Redis Version: 3.2.4 (from AWS ElastiCache)

Redis connect with socket.io-redis

let Adapter = adapter({host: config.redis.host, port: config.redis.port});
io.adapter(Adapter);

According to source of socket.io-redis, I found node_redis, that was redis client.

@mavrick
Copy link

mavrick commented Jan 16, 2017

@darrachequesne doesn't msgpack get called inside the publish method? I don't think I should have to call msgpack myself, or do I?

What version did you rollback to @DingGGu ? I may have to do the same at this stage.

@darrachequesne
Copy link
Member

redisPub.publish("discord", JSON.stringify({type: "status", data: {uuid: id, status: status}}));

@mavrick I may be missing something here, but if you want to publish some packets, why aren't you using the socket.io-emitter directly, instead of the redis pub?

var io = require('socket.io-emitter')({ host: '127.0.0.1', port: 6379 });
io.emit("discord", JSON.stringify({type: "status", data: {uuid: id, status: status}}));
// msgpack.encode gets indeed called

@mavrick
Copy link

mavrick commented Jan 16, 2017

@darrachequesne i can't argue with that logic. I'll give it a go this evening and see what results I get.

Cheers

@DingGGu
Copy link
Contributor Author

DingGGu commented Jan 18, 2017

@mavrick Here is my versions.

"socket.io-emitter": "^1.0.0",
"socket.io-redis": "^2.0.1"

I guess maybe socket.io-emitter has same issue too.
I downgraded two both of packages.

@mavrick
Copy link

mavrick commented Jan 18, 2017

After using socket.io-emitter instead of redis.publish() my msgpack-lite errors have gone away. Will do some more testing over the coming days.

@darrachequesne
Copy link
Member

@mavrick thanks for the feedback!

"socket.io-emitter": "^1.0.0",
"socket.io-redis": "^2.0.1"

@DingGGu those version both use msgpack-js, so you think that is related to the switch to msgpack-lite in the newer versions (socket.io-emitter 2.x and socket.io-redis 3.x)?

@DingGGu
Copy link
Contributor Author

DingGGu commented Jan 20, 2017

@darrachequesne Yes,
Our service is chatting service with 1000+ users.
Most users said after update, It relates msgpack-lite bugs, "Sometimes, a few message was lost in my connection and can't receive anymore."

@jeroenrinzema
Copy link

jeroenrinzema commented Jan 24, 2017

I get the same error i dived into socket.io-redis and found that i can encode and decode my payload gets correctly encoded with a length of 85 characters but when the message get's received is the encoded message 84 characters long.

The payload:

[ 'ATcq1N', { type: 2, data: [ 'hi' ], nsp: '/' }, { except: [ 'qeoUFZeGyBlewzL_AAAA' ], rooms: undefined, flags: { broadcast: true } } ]

example

line where message get's encoded

line where message get's decoded

@jeroenrinzema
Copy link

jeroenrinzema commented Jan 24, 2017

msgpack is the problem. I think that the encoded message is somewhere parsed through .toString() this results that in the encoded message sometimes data gets lost. Below did i included the same message with the second one after encoding parsed through .toString(). You see that \u0002 disappears.

Not parsed
��YMf4EB��type\u0002�data��hi�nsp�/��except��BoOzf02kzKgSjGSuAAAA�roomsĥflags��broadcast�

Parsed
��YMf4EB��type�data��hi�nsp�/��except��BoOzf02kzKgSjGSuAAAA�roomsĥflags��broadcast�

A simple but maybe dirty and temporarily fix is to replace msgpack with JSON (stringify | parse) this resolves the problem.

@darrachequesne
Copy link
Member

The behaviour you are describing looks really weird.

  • Which .toString() method are you referring to?
  • Where does the "parsed" section come from? Is that the msg argument in the onmessage method?
  • Is it linked to a given Redis version? (as suggested here)

The problem with that fix is that JSON.stringify/parse does not handle binary message, we could rollback to using msgpack-js, or any other msgpack implementation, yet there's no guarantee it would fix the issue, since we are not able to reproduce (or are we?).

@jeroenrinzema
Copy link

  • I am referring to JavaScript String toString() Method
  • Sorry i was maybe not very clear. I took twice the same encoded message and modified one of the messages with the String toString() Method as described above. What i found was that the modified message lost a character in the process. You can see this happening if you log the string lengths of the message after the message get's encoded and before it get's decoded somewhere get's data lost while sending over Redis.
  • I am running Redis server v=3.2.6 sha=00000000:0 malloc=libc bits=64 build=8eda9288163b093b i don't know if this problem get's resolved on other Redis versions.

Yes that is maybe not a bad idea but i found that in msgpack-js the same issue exists. I did the same test with msgpack and got good results. I don't know if we are able to reproduce the problem i can try to set up a test environment later today trying to figure out if i am able to recreate the problem.

@emhagman
Copy link

emhagman commented Mar 29, 2017

Redis Version: 3.2.4
Socket.io-redis Version: 4.0.0

I have also experienced this exact issue with the message:

/app/node_modules/msgpack-js/msgpack.js:195 
throw new Error("Unknown type 0x" + type.toString(16)); 
^ 
Error: Unknown type 0xc7 
at Decoder.parse (/app/node_modules/msgpack-js/msgpack.js:195:9) 
at Decoder.map (/app/node_modules/msgpack-js/msgpack.js:34:23) 
at Decoder.array (/app/node_modules/msgpack-js/msgpack.js:51:21) 
at Decoder.map (/app/node_modules/msgpack-js/msgpack.js:34:23) 
at Decoder.array (/app/node_modules/msgpack-js/msgpack.js:51:21) 
at Decoder.map (/app/node_modules/msgpack-js/msgpack.js:34:23) 
at Decoder.array (/app/node_modules/msgpack-js/msgpack.js:51:21) 
at Object.decode (/app/node_modules/msgpack-js/msgpack.js:199:23) 
at Redis.onmessage (/app/node_modules/socket.io-redis/index.js:134:24) 
at emitTwo (events.js:106:13) 
at RedisClient.emit (events.js:191:7) 
at return_pub_sub (/app/node_modules/socket.io-redis/node_modules/redis/index.js:778:18) 
at RedisClient.return_reply (/app/node_modules/socket.io-redis/node_modules/redis/index.js:825:9) 
at JavascriptRedisParser.returnReply (/app/node_modules/socket.io-redis/node_modules/redis/index.js:188:18) 
at JavascriptRedisParser.execute (/app/node_modules/redis-parser/lib/parser.js:562:12) 
at Socket.<anonymous> (/app/node_modules/socket.io-redis/node_modules/redis/index.js:267:27) 
at emitOne (events.js:96:13) 
at Socket.emit (events.js:188:7) 
at readableAddChunk (_stream_readable.js:176:18) 
at TCP.onread [as _originalOnread] (net.js:548:20) 
at TCP.onread (/app/node_modules/async-listener/glue.js:188:31) 

@darrachequesne
Copy link
Member

@emhagman msgpack-js was replaced by msgpack-lite (#156), are you sure you're using socket.io-redis 4.0.0 (since the stacktrace mentions msgpack-js)?

@DingGGu
Copy link
Contributor Author

DingGGu commented Apr 4, 2017

👍

@Piper0804
Copy link

Downgrading "socket.io-redis" to "^2.0.1" solved this same issue for us as well.

@darrachequesne
Copy link
Member

Closed by #218.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants