Skip to content

Loading…

storing large objects #6

Closed
jeroenooms opened this Issue · 5 comments

3 participants

@jeroenooms

rredis seems to be ok for small sized, data, but when I try to store a large object the connection seems to die:

> redisSet("test", rnorm(1e8))
Error in doTryCatch(return(expr), name, parentenv, handler) : 
  ERR Protocol error: invalid bulk length
In addition: Warning messages:
1: In writeBin(v, con) : problem writing to connection
2: In writeBin(.raw("\r\n"), con) : problem writing to connection

Any idea what is going on?

@bwlewis
Owner
@userbc

I am having the same problem as jeroenooms. Did you ever manage to find a solution to this or is it still unsolved?

Thanks

@bwlewis
Owner

Hi! Redis values are limited to 512MB, see for example:

http://redis.io/topics/data-types

Any R object exceeding this length will result in an invalid bulk length error.

There was a bug in Redis affecting HMSET that could result in this kind of error happening randomly. it was fixed months ago. Here is a reference: antirez/redis#673

The upshot is, if you need to store large objects in redis, you'll have to compile a custom redis server.

@bwlewis bwlewis closed this
@userbc

Thank you. You say that to store objects larger than 512MB I need to compile a custom redis server, is this something I can change in the redis.conf file? Can you point me in the direction of what I need to change please?

@bwlewis
Owner

This is actually an open bug in Redis. Unfortunately, it's not a configuration option. And it appears that even changing the source code of Redis to support this is trickier than I thought. See the following bug thread for a discussion on this topic:

antirez/redis#757

So it looks like 512MB will be the limit for a while yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.