Add bitwise operations for integer types #5149

tpoterba · 2019-01-16T02:20:45Z

No description provided.

danking

This is gonna be so awesome. Once Daniel and I get an arbitrary cross product working, this plus aggregators will give us a path to very tight representations of variant sample matrix!

danking · 2019-01-16T14:55:28Z

hail/python/hail/expr/functions.py

+
+@typecheck(x=expr_oneof(expr_int32, expr_int64), y=expr_int32)
+def bit_rshift(x, y):
+    """Bitwise right-shift `x` by `y`.


I think we should document that this is a sign extending bitwise right shift.

(which is also what python means by >>, so we're good, I just think we should be explicit)

Moreover, I think python elides a logical right shift operator (called >>> in Java/Scala) because Python ints are arbitrary width, ergo, it's maybe not clear where the high zeros are showing up. We (hail), have fixed-width integers so a logical right shift operator can be sensibly defined.

Can we add bit_logical_rshift(x, y) as well? Or add a logical keyword argument? Or maybe preserve_sign?

@danking Thanks for doing the work for me ;)

added logical argument and docs

danking · 2019-01-16T15:06:16Z

Also, it looks like we didn't define the operator syntax. Sounds like an easy PR to farm out to someone else!

jigold · 2019-01-16T14:36:47Z

hail/python/hail/expr/functions.py

+    32
+
+    Notes
+    -----


Do we need two different right shift operations (sign extended version and not sign extended)? Add a note saying which version we've implemented.

jigold · 2019-01-16T14:43:40Z

hail/python/hail/expr/functions.py

+
+
+@typecheck(x=expr_oneof(expr_int32, expr_int64), y=expr_oneof(expr_int32, expr_int64))
+def bit_xor(x, y):


This is duplicated from above.

jigold · 2019-01-16T14:58:38Z

hail/python/hail/expr/functions.py

+
+    >>> hl.eval(hl.bit_lshift(1, 8))
+    256
+


How does our code behave for the following case? 1 << 32

The Python interpreter is this:

>>> 1 << 32 4294967296

The Scala interpreter is this:

scala> 1 << 32 res6: Int = 1

It looks like Scala wraps the bits being shifted and Python uses a long integer with arbitrary size.

Python:

>>> 1 << 64 18446744073709551616 >>> 1 << 65 36893488147419103232 >>> 1 << 1000 10715086071862673209484250490600018105614048117055336074437503883703510511249361224931983788156958581275946729175531468251871452856923140435984577574698574803934567774824230985421074605062371141877954182153046474983581941267398767559165543946077062914571196477686542167660429831652624386837205668069376

Scala:

scala> 1 << 64 res7: Int = 1 scala> 1 << 65 res8: Int = 2

If we're using Scala bit operations, then we need to make the behavior difference from Python clear in the docs.

Yup, clarified. Good point.

jigold · 2019-01-16T15:02:28Z

hail/src/main/scala/is/hail/expr/ir/BinaryOp.scala

      case (_: TInt64, _: TInt64) =>
        val ll = coerce[Long](l)
+


jigold · 2019-01-16T15:06:45Z

hail/src/main/scala/is/hail/expr/ir/BinaryOp.scala

@@ -93,6 +110,10 @@ object BinaryOp {
    case "*" | "Multiply" => Multiply()
    case "/" | "FloatingPointDivide"  => FloatingPointDivide()
    case "//" | "RoundToNegInfDivide" => RoundToNegInfDivide()
+    case "|" | "BitOr" => BitOr()
+    case "&" | "BitAnd" => BitAnd()
+    case "<<" | "LeftShift" => LeftShift()


Are you missing BitXOr here?

yep, tests caught it too

jigold · 2019-01-16T15:07:42Z

hail/src/test/scala/is/hail/expr/ir/IRSuite.scala

+    assertEvalsTo(ApplyUnaryPrimOp(BitFlip(), i32na), null)
+    assertEvalsTo(ApplyUnaryPrimOp(BitFlip(), I64(0xdeadbeef12345678L)), ~0xdeadbeef12345678L)
+    assertEvalsTo(ApplyUnaryPrimOp(BitFlip(), i64na), null)
+


jigold · 2019-01-16T15:08:18Z

hail/src/test/scala/is/hail/expr/ir/IRSuite.scala

+    assertEvalsTo(ApplyBinaryPrimOp(LeftShift(), i32na, I32(2)), null)
+    assertEvalsTo(ApplyBinaryPrimOp(LeftShift(), i32na, i32na), null)
+
+    assertEvalsTo(ApplyBinaryPrimOp(LeftShift(), I64(5), I32(2)), 5L << 2)


Can you add tests with more edge cases and negative numbers?

cseed · 2019-01-16T15:21:09Z

hail/python/hail/docs/functions/index.rst

+    bit_xor
+    bit_lshift
+    bit_rshift
+    bit_flip


Nice. I think bit_not is more traditional. I also agree we should have logical and arithmetic right shifts.

tpoterba · 2019-01-16T16:06:22Z

Also, it looks like we didn't define the operator syntax. Sounds like an easy PR to farm out to someone else!

@danking I intentionally didn't add this yet - there is a case I am worried about:

mt.filter_rows(mt.pass & mt.variant_qc.AF[1] > 0.01)

Right now this is a type error. With bit operators, this is the same as:

mt.filter_rows((hl.bit_and(hl.int(mt.pass), mt.variant_qc.AF[1]) > 0.01)

danking · 2019-01-16T16:53:49Z

@tpoterba looks like you've written a very popular PR ;)

Strong agree on the operator stuff. Eck, that's really a mess.

tpoterba · 2019-01-16T17:16:14Z

Good comments. Addressed:

renamed bit_flip to bit_not
added logical flag to bit_rshift
added more tests with negative numbers
expanded docs around differences between Python and Hail

x

jigold · 2019-01-16T18:00:21Z

hail/src/test/scala/is/hail/expr/ir/IRSuite.scala

    assertEvalsTo(ApplyBinaryPrimOp(LeftShift(), I32(5), i32na), null)
    assertEvalsTo(ApplyBinaryPrimOp(LeftShift(), i32na, I32(2)), null)
    assertEvalsTo(ApplyBinaryPrimOp(LeftShift(), i32na, i32na), null)

    assertEvalsTo(ApplyBinaryPrimOp(LeftShift(), I64(5), I32(2)), 5L << 2)
+    assertEvalsTo(ApplyBinaryPrimOp(LeftShift(), I64(-5), I64(2)), -5L<< 2)


jigold · 2019-01-16T18:00:42Z

hail/src/test/scala/is/hail/expr/ir/IRSuite.scala

    assertEvalsTo(ApplyBinaryPrimOp(RightShift(), I32(5), i32na), null)
    assertEvalsTo(ApplyBinaryPrimOp(RightShift(), i32na, I32(2)), null)
    assertEvalsTo(ApplyBinaryPrimOp(RightShift(), i32na, i32na), null)

    assertEvalsTo(ApplyBinaryPrimOp(RightShift(), I64(0xffff5), I32(2)), 0xffff5L >> 2)
+    assertEvalsTo(ApplyBinaryPrimOp(RightShift(), I64(-5), I64(2)), -5L>> 2)


jigold · 2019-01-16T19:04:17Z

hail/src/test/scala/is/hail/expr/ir/IRSuite.scala

    assertEvalsTo(ApplyBinaryPrimOp(BitXOr(), I32(5), i32na), null)
    assertEvalsTo(ApplyBinaryPrimOp(BitXOr(), i32na, I32(2)), null)
    assertEvalsTo(ApplyBinaryPrimOp(BitXOr(), i32na, i32na), null)

    assertEvalsTo(ApplyBinaryPrimOp(BitXOr(), I64(5), I64(2)), 5L ^ 2L)
+    assertEvalsTo(ApplyBinaryPrimOp(BitOr(), I64(-5), I64(2)), -5L ^ 2L)


Need to be BitXOr (typos)

jigold · 2019-01-16T19:04:36Z

hail/src/test/scala/is/hail/expr/ir/IRSuite.scala

+    assertEvalsTo(ApplyBinaryPrimOp(LogicalRightShift(), i32na, i32na), null)
+
+    assertEvalsTo(ApplyBinaryPrimOp(LogicalRightShift(), I64(0xffff5), I32(2)), 0xffff5L >>> 2)
+    assertEvalsTo(ApplyBinaryPrimOp(LogicalRightShift(), I64(-5), I64(2)), -5L>>> 2)


addressed

tpoterba · 2019-01-17T20:45:21Z

ready for another review

Aadd bitwise operations for integer types

2355b61

tpoterba added the enhancement label Jan 16, 2019

tpoterba assigned jigold Jan 16, 2019

fix typo

4102937

danking previously requested changes Jan 16, 2019

View reviewed changes

jigold previously requested changes Jan 16, 2019

View reviewed changes

cseed reviewed Jan 16, 2019

View reviewed changes

Addressed comments

11c5bf5

jigold previously requested changes Jan 16, 2019

View reviewed changes

tpoterba added 2 commits January 16, 2019 14:29

fix spacing

06186c6

fix bitxor

a243321

fix logical right shift, add tests

cca397b

jigold approved these changes Jan 17, 2019

View reviewed changes

tpoterba added 2 commits January 18, 2019 13:28

fix test

22db9f7

fix parens

6ea4e9b

danking merged commit 2e4cded into hail-is:master Jan 19, 2019

This was referenced Apr 29, 2024

Hail 0.2.10 patch notes iris-garden/test-process#1662

Closed

Hail 0.2.10 patch notes iris-garden/test-process#2248

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add bitwise operations for integer types #5149

Add bitwise operations for integer types #5149

tpoterba commented Jan 16, 2019

danking left a comment

danking Jan 16, 2019

danking Jan 16, 2019

danking Jan 16, 2019

jigold Jan 16, 2019

tpoterba Jan 16, 2019

danking commented Jan 16, 2019

jigold Jan 16, 2019

jigold Jan 16, 2019

jigold Jan 16, 2019

tpoterba Jan 16, 2019

jigold Jan 16, 2019

jigold Jan 16, 2019

tpoterba Jan 16, 2019

jigold Jan 16, 2019

jigold Jan 16, 2019

cseed Jan 16, 2019

tpoterba commented Jan 16, 2019

danking commented Jan 16, 2019

tpoterba commented Jan 16, 2019

jigold Jan 16, 2019

jigold Jan 16, 2019

jigold Jan 16, 2019

jigold Jan 16, 2019

tpoterba commented Jan 17, 2019



		@typecheck(x=expr_oneof(expr_int32, expr_int64), y=expr_oneof(expr_int32, expr_int64))
		def bit_xor(x, y):

Add bitwise operations for integer types #5149

Add bitwise operations for integer types #5149

Conversation

tpoterba commented Jan 16, 2019

danking left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danking commented Jan 16, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tpoterba commented Jan 16, 2019

danking commented Jan 16, 2019

tpoterba commented Jan 16, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tpoterba commented Jan 17, 2019