New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Potential problem with setbit and getbit when used with BitSet.valueOf() #301

Closed
gmuller opened this Issue May 20, 2012 · 6 comments

Comments

Projects
None yet
4 participants
@gmuller

gmuller commented May 20, 2012

While experimenting with the code presented in this very popular blog post related to redis bitmaps, I came across a problem that I can't seem to get my head around.

It seems that when you individually write bits to redis via the Jedis setbit command, everything works properly as long as you use the getbit command to get the bit status back out. If you read the entire key however, the representation of the bitset appears to be in every way linear, and couldn't possibly work with BitSet.valueOf().

Here is a gist that inserts and retrieves the bitset in various ways, none of them work as expected.

It looks as though the get command retrieves that byte[] in literal order. For instance if I write all of the primes to 20 as a bit set into Redis, then read the array out, I get a byte[] representation of:

  • byte[0] - 53
  • byte[1] - 20
  • byte[2] - 80

If I look at the binary representation of the first byte, I see 00110101. If read from left to right, like a human would, I see that bit position 2, 3, 5, and 7 are set, corresponding to the correct primes. Read, right to left, as BitSet.valueOf() would read them, the values are of course all wrong.

Additionally, when read via BitSet.valueOf() the byte order is backwards. I tried reversing the byte order, which would work just fine if the bit positions are read in reverse order.

This may be expected behavior, but the renders the code in the blog post referenced above incorrect. Given the code that I've tried reading the value of the key from Redis using Jedis and BitSet.valueOf() will never work properly without intermediary methods.

Additional notes:
Platform: Mac OSX Lion (7.4)
Redis 2.4.13 installed via Homebrew
Java 7
Jedis 2.1.0

@xetorthio

This comment has been minimized.

Show comment
Hide comment
@xetorthio

xetorthio May 21, 2012

Owner

Hi!
That depends on how you store the bits. If the problem is that BitSet reads
from right to left, you should store the bits from right to left.
Either SETBIT and GETBIT commands receives the position of the bit you want
to set or get.

Jonathan

On Sun, May 20, 2012 at 1:48 PM, Grant Muller <
reply@reply.github.com

wrote:

While experimenting with the code presented in this very popular blog
post

related to redis bitmaps, I came across a problem that I can't seem to get
my head around.

It seems that when you individually write bits to redis via the Jedis
setbit command, everything works properly as long as you use the getbit
command to get the bit status back out. If you read the entire key however,
the representation of the bitset appears to be in every way linear, and
couldn't possibly work with BitSet.valueOf().

Here is a gist that inserts and
retrieves the bitset in various ways, none of them work as expected.

It looks as though the get command retrieves that byte[] in literal order.
For instance if I write all of the primes to 20 as a bit set into Redis,
then read the array out, I get a byte[] representation of:

  • byte[0] - 53
  • byte[1] - 20
  • byte[2] - 80

If I look at the binary representation of the first byte, I see 00110101.
If read from left to right, like a human would, I see that bit position 2,
3, 5, and 7 are set, corresponding to the correct primes. Read, right to
left, as BitSet.valueOf() would read them, the values are of course all
wrong.

Additionally, when read via BitSet.valueOf() the byte order is backwards.
I tried reversing the byte order, which would work just fine if the bit
positions are read in reverse order.

This may be expected behavior, but the renders the code in the blog post
referenced above incorrect. Given the code that I've tried reading the
value of the key from Redis using Jedis and BitSet.valueOf() will never
work properly without intermediary methods.

Additional notes:
Platform: Mac OSX Lion (7.4)
Redis 2.4.13 installed via Homebrew
Java 7
Jedis 2.1.0


Reply to this email directly or view it on GitHub:
#301

Owner

xetorthio commented May 21, 2012

Hi!
That depends on how you store the bits. If the problem is that BitSet reads
from right to left, you should store the bits from right to left.
Either SETBIT and GETBIT commands receives the position of the bit you want
to set or get.

Jonathan

On Sun, May 20, 2012 at 1:48 PM, Grant Muller <
reply@reply.github.com

wrote:

While experimenting with the code presented in this very popular blog
post

related to redis bitmaps, I came across a problem that I can't seem to get
my head around.

It seems that when you individually write bits to redis via the Jedis
setbit command, everything works properly as long as you use the getbit
command to get the bit status back out. If you read the entire key however,
the representation of the bitset appears to be in every way linear, and
couldn't possibly work with BitSet.valueOf().

Here is a gist that inserts and
retrieves the bitset in various ways, none of them work as expected.

It looks as though the get command retrieves that byte[] in literal order.
For instance if I write all of the primes to 20 as a bit set into Redis,
then read the array out, I get a byte[] representation of:

  • byte[0] - 53
  • byte[1] - 20
  • byte[2] - 80

If I look at the binary representation of the first byte, I see 00110101.
If read from left to right, like a human would, I see that bit position 2,
3, 5, and 7 are set, corresponding to the correct primes. Read, right to
left, as BitSet.valueOf() would read them, the values are of course all
wrong.

Additionally, when read via BitSet.valueOf() the byte order is backwards.
I tried reversing the byte order, which would work just fine if the bit
positions are read in reverse order.

This may be expected behavior, but the renders the code in the blog post
referenced above incorrect. Given the code that I've tried reading the
value of the key from Redis using Jedis and BitSet.valueOf() will never
work properly without intermediary methods.

Additional notes:
Platform: Mac OSX Lion (7.4)
Redis 2.4.13 installed via Homebrew
Java 7
Jedis 2.1.0


Reply to this email directly or view it on GitHub:
#301

@gmuller

This comment has been minimized.

Show comment
Hide comment
@gmuller

gmuller May 22, 2012

but wouldn't that prevent it from being cross-platform. If I

  1. jedis.setbit(key, 2, true)
  2. redis-cli GETBIT 2 (WORKS)
  3. BItSet.valueOf(jedis.get(key)) (DOESN'T WORK)

And if I do this:

  1. Create BitSet with bit 2 set
  2. jedis.set(key, bitset)
  3. redis-cli GETBIT 2 (DOESN'T RETURN CORRECT STATUS)
  4. BItSet(jedis.get(key)) (WORKS)

This all may be expected behavior, it just implies that you cannot use BitSet.valueOf in java out of the box without reforming the order of the bits and bytes that you're reading.

gmuller commented May 22, 2012

but wouldn't that prevent it from being cross-platform. If I

  1. jedis.setbit(key, 2, true)
  2. redis-cli GETBIT 2 (WORKS)
  3. BItSet.valueOf(jedis.get(key)) (DOESN'T WORK)

And if I do this:

  1. Create BitSet with bit 2 set
  2. jedis.set(key, bitset)
  3. redis-cli GETBIT 2 (DOESN'T RETURN CORRECT STATUS)
  4. BItSet(jedis.get(key)) (WORKS)

This all may be expected behavior, it just implies that you cannot use BitSet.valueOf in java out of the box without reforming the order of the bits and bytes that you're reading.

@xetorthio

This comment has been minimized.

Show comment
Hide comment
@xetorthio

xetorthio May 22, 2012

Owner

How do you exactly do:
jedis.set(key, bitset)?

There is no overload for BitSet in Jedis. Actually Jedis doesn't know
anything about BitSets... you have to tell him how to serialize a BitSet in
Redis and how to deserialize it later.

On Mon, May 21, 2012 at 10:55 PM, Grant Muller <
reply@reply.github.com

wrote:

but wouldn't that prevent it from being cross-platform. If I

  1. jedis.setbit(key, 2, true)
  2. redis-cli GETBIT 2 (WORKS)
  3. BItSet.valueOf(jedis.get(key)) (DOESN'T WORK)

And if I do this:

  1. Create BitSet with bit 2 set
  2. jedis.set(key, bitset)
  3. redis-cli GETBIT 2 (DOESN'T RETURN CORRECT STATUS)
  4. BItSet(jedis.get(key)) (WORKS)

This all may be expected behavior, it just implies that you cannot use
BitSet.valueOf in java out of the box without reforming the order of the
bits and bytes that you're reading.


Reply to this email directly or view it on GitHub:
#301 (comment)

Owner

xetorthio commented May 22, 2012

How do you exactly do:
jedis.set(key, bitset)?

There is no overload for BitSet in Jedis. Actually Jedis doesn't know
anything about BitSets... you have to tell him how to serialize a BitSet in
Redis and how to deserialize it later.

On Mon, May 21, 2012 at 10:55 PM, Grant Muller <
reply@reply.github.com

wrote:

but wouldn't that prevent it from being cross-platform. If I

  1. jedis.setbit(key, 2, true)
  2. redis-cli GETBIT 2 (WORKS)
  3. BItSet.valueOf(jedis.get(key)) (DOESN'T WORK)

And if I do this:

  1. Create BitSet with bit 2 set
  2. jedis.set(key, bitset)
  3. redis-cli GETBIT 2 (DOESN'T RETURN CORRECT STATUS)
  4. BItSet(jedis.get(key)) (WORKS)

This all may be expected behavior, it just implies that you cannot use
BitSet.valueOf in java out of the box without reforming the order of the
bits and bytes that you're reading.


Reply to this email directly or view it on GitHub:
#301 (comment)

@gresrun

This comment has been minimized.

Show comment
Hide comment
@gresrun

gresrun May 23, 2012

It looks to me like the issue revolves around the fact that there is not a clear, documented way to get the values of a Redis bitset into a java.util.BitSet.

There are at least 4 methods that I could think of to do this, most of which don't do what you want.

In order to get the right answer, I had to reverse the order of the bits in the bytes returned from final byte[] sieveKeyBytes = redis.get(sieveSetKey.getBytes());. See https://gist.github.com/2773136 for my exploration of this.

Jonathan, does it make sense to you to add a utility method to help people out with this?

gresrun commented May 23, 2012

It looks to me like the issue revolves around the fact that there is not a clear, documented way to get the values of a Redis bitset into a java.util.BitSet.

There are at least 4 methods that I could think of to do this, most of which don't do what you want.

In order to get the right answer, I had to reverse the order of the bits in the bytes returned from final byte[] sieveKeyBytes = redis.get(sieveSetKey.getBytes());. See https://gist.github.com/2773136 for my exploration of this.

Jonathan, does it make sense to you to add a utility method to help people out with this?

@gmuller

This comment has been minimized.

Show comment
Hide comment
@gmuller

gmuller May 23, 2012

Greg has it exactly right. Again, this isn't a requirement (but it would be nice to have a helper method built in). All I was trying to point out was that the built in methods aren't going to work, and to make sure that it wasn't because a bug in the way that the bytestream was being returned. It didn't look like it, its just the way that Redis stores the array (in VERY literal order).

Thanks for looking into it a little deeper guys. I'll keep a look out here for a helper method just in case. Otherwise referencing the gist Greg created will work for others doing this on their own.

gmuller commented May 23, 2012

Greg has it exactly right. Again, this isn't a requirement (but it would be nice to have a helper method built in). All I was trying to point out was that the built in methods aren't going to work, and to make sure that it wasn't because a bug in the way that the bytestream was being returned. It didn't look like it, its just the way that Redis stores the array (in VERY literal order).

Thanks for looking into it a little deeper guys. I'll keep a look out here for a helper method just in case. Otherwise referencing the gist Greg created will work for others doing this on their own.

@gmuller

This comment has been minimized.

Show comment
Hide comment
@gmuller

gmuller Jun 15, 2012

I updated the gist to include methods for going to and from BitSet to byte[] properly. I'm not all that familiar with the way that Jedis is laid out, otherwise I'd fork and add them myself. https://gist.github.com/2933940

gmuller commented Jun 15, 2012

I updated the gist to include methods for going to and from BitSet to byte[] properly. I'm not all that familiar with the way that Jedis is laid out, otherwise I'd fork and add them myself. https://gist.github.com/2933940

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment