Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add bitwise operations for integer types #5149

Merged
merged 8 commits into from Jan 19, 2019

Conversation

Projects
None yet
5 participants
@tpoterba
Copy link
Collaborator

commented Jan 16, 2019

No description provided.

@chrisvittal

This comment has been minimized.

Copy link
Collaborator

commented on hail/python/hail/expr/functions.py in 2355b61 Jan 16, 2019

Typo

@danking
Copy link
Collaborator

left a comment

This is gonna be so awesome. Once Daniel and I get an arbitrary cross product working, this plus aggregators will give us a path to very tight representations of variant sample matrix!


@typecheck(x=expr_oneof(expr_int32, expr_int64), y=expr_int32)
def bit_rshift(x, y):
"""Bitwise right-shift `x` by `y`.

This comment has been minimized.

Copy link
@danking

danking Jan 16, 2019

Collaborator

I think we should document that this is a sign extending bitwise right shift.

This comment has been minimized.

Copy link
@danking

danking Jan 16, 2019

Collaborator

(which is also what python means by >>, so we're good, I just think we should be explicit)

This comment has been minimized.

Copy link
@danking

danking Jan 16, 2019

Collaborator

Moreover, I think python elides a logical right shift operator (called >>> in Java/Scala) because Python ints are arbitrary width, ergo, it's maybe not clear where the high zeros are showing up. We (hail), have fixed-width integers so a logical right shift operator can be sensibly defined.

Can we add bit_logical_rshift(x, y) as well? Or add a logical keyword argument? Or maybe preserve_sign?

This comment has been minimized.

Copy link
@jigold

jigold Jan 16, 2019

Collaborator

@danking Thanks for doing the work for me ;)

This comment has been minimized.

Copy link
@tpoterba

tpoterba Jan 16, 2019

Author Collaborator

added logical argument and docs

@danking

This comment has been minimized.

Copy link
Collaborator

commented Jan 16, 2019

Also, it looks like we didn't define the operator syntax. Sounds like an easy PR to farm out to someone else!

32
Notes
-----

This comment has been minimized.

Copy link
@jigold

jigold Jan 16, 2019

Collaborator

Do we need two different right shift operations (sign extended version and not sign extended)? Add a note saying which version we've implemented.



@typecheck(x=expr_oneof(expr_int32, expr_int64), y=expr_oneof(expr_int32, expr_int64))
def bit_xor(x, y):

This comment has been minimized.

Copy link
@jigold

jigold Jan 16, 2019

Collaborator

This is duplicated from above.

>>> hl.eval(hl.bit_lshift(1, 8))
256

This comment has been minimized.

Copy link
@jigold

jigold Jan 16, 2019

Collaborator

How does our code behave for the following case? 1 << 32

The Python interpreter is this:

>>> 1 << 32
4294967296

The Scala interpreter is this:

scala> 1 << 32
res6: Int = 1

It looks like Scala wraps the bits being shifted and Python uses a long integer with arbitrary size.

Python:

>>> 1 << 64
18446744073709551616
>>> 1 << 65
36893488147419103232
>>> 1 << 1000
10715086071862673209484250490600018105614048117055336074437503883703510511249361224931983788156958581275946729175531468251871452856923140435984577574698574803934567774824230985421074605062371141877954182153046474983581941267398767559165543946077062914571196477686542167660429831652624386837205668069376

Scala:

scala> 1 << 64
res7: Int = 1
scala> 1 << 65
res8: Int = 2

If we're using Scala bit operations, then we need to make the behavior difference from Python clear in the docs.

This comment has been minimized.

Copy link
@tpoterba

tpoterba Jan 16, 2019

Author Collaborator

Yup, clarified. Good point.

case (_: TInt64, _: TInt64) =>
val ll = coerce[Long](l)

This comment has been minimized.

Copy link
@jigold

jigold Jan 16, 2019

Collaborator

delete.

@@ -93,6 +110,10 @@ object BinaryOp {
case "*" | "Multiply" => Multiply()
case "/" | "FloatingPointDivide" => FloatingPointDivide()
case "//" | "RoundToNegInfDivide" => RoundToNegInfDivide()
case "|" | "BitOr" => BitOr()
case "&" | "BitAnd" => BitAnd()
case "<<" | "LeftShift" => LeftShift()

This comment has been minimized.

Copy link
@jigold

jigold Jan 16, 2019

Collaborator

Are you missing BitXOr here?

This comment has been minimized.

Copy link
@tpoterba

tpoterba Jan 16, 2019

Author Collaborator

yep, tests caught it too

assertEvalsTo(ApplyUnaryPrimOp(BitFlip(), i32na), null)
assertEvalsTo(ApplyUnaryPrimOp(BitFlip(), I64(0xdeadbeef12345678L)), ~0xdeadbeef12345678L)
assertEvalsTo(ApplyUnaryPrimOp(BitFlip(), i64na), null)

This comment has been minimized.

Copy link
@jigold

jigold Jan 16, 2019

Collaborator

delete.

assertEvalsTo(ApplyBinaryPrimOp(LeftShift(), i32na, I32(2)), null)
assertEvalsTo(ApplyBinaryPrimOp(LeftShift(), i32na, i32na), null)

assertEvalsTo(ApplyBinaryPrimOp(LeftShift(), I64(5), I32(2)), 5L << 2)

This comment has been minimized.

Copy link
@jigold

jigold Jan 16, 2019

Collaborator

Can you add tests with more edge cases and negative numbers?

bit_xor
bit_lshift
bit_rshift
bit_flip

This comment has been minimized.

Copy link
@cseed

cseed Jan 16, 2019

Collaborator

Nice. I think bit_not is more traditional. I also agree we should have logical and arithmetic right shifts.

@tpoterba

This comment has been minimized.

Copy link
Collaborator Author

commented Jan 16, 2019

Also, it looks like we didn't define the operator syntax. Sounds like an easy PR to farm out to someone else!

@danking I intentionally didn't add this yet - there is a case I am worried about:

mt.filter_rows(mt.pass & mt.variant_qc.AF[1] > 0.01)

Right now this is a type error. With bit operators, this is the same as:

mt.filter_rows((hl.bit_and(hl.int(mt.pass), mt.variant_qc.AF[1]) > 0.01)
@danking

This comment has been minimized.

Copy link
Collaborator

commented Jan 16, 2019

@tpoterba looks like you've written a very popular PR ;)

Strong agree on the operator stuff. Eck, that's really a mess.

@tpoterba

This comment has been minimized.

Copy link
Collaborator Author

commented Jan 16, 2019

Good comments. Addressed:

  • renamed bit_flip to bit_not
  • added logical flag to bit_rshift
  • added more tests with negative numbers
  • expanded docs around differences between Python and Hail

@tpoterba tpoterba dismissed stale reviews from jigold and danking Jan 16, 2019

x

assertEvalsTo(ApplyBinaryPrimOp(LeftShift(), I32(5), i32na), null)
assertEvalsTo(ApplyBinaryPrimOp(LeftShift(), i32na, I32(2)), null)
assertEvalsTo(ApplyBinaryPrimOp(LeftShift(), i32na, i32na), null)

assertEvalsTo(ApplyBinaryPrimOp(LeftShift(), I64(5), I32(2)), 5L << 2)
assertEvalsTo(ApplyBinaryPrimOp(LeftShift(), I64(-5), I64(2)), -5L<< 2)

This comment has been minimized.

Copy link
@jigold

jigold Jan 16, 2019

Collaborator

Spacing.

assertEvalsTo(ApplyBinaryPrimOp(RightShift(), I32(5), i32na), null)
assertEvalsTo(ApplyBinaryPrimOp(RightShift(), i32na, I32(2)), null)
assertEvalsTo(ApplyBinaryPrimOp(RightShift(), i32na, i32na), null)

assertEvalsTo(ApplyBinaryPrimOp(RightShift(), I64(0xffff5), I32(2)), 0xffff5L >> 2)
assertEvalsTo(ApplyBinaryPrimOp(RightShift(), I64(-5), I64(2)), -5L>> 2)

This comment has been minimized.

Copy link
@jigold

jigold Jan 16, 2019

Collaborator

Spacing

assertEvalsTo(ApplyBinaryPrimOp(BitXOr(), I32(5), i32na), null)
assertEvalsTo(ApplyBinaryPrimOp(BitXOr(), i32na, I32(2)), null)
assertEvalsTo(ApplyBinaryPrimOp(BitXOr(), i32na, i32na), null)

assertEvalsTo(ApplyBinaryPrimOp(BitXOr(), I64(5), I64(2)), 5L ^ 2L)
assertEvalsTo(ApplyBinaryPrimOp(BitOr(), I64(-5), I64(2)), -5L ^ 2L)

This comment has been minimized.

Copy link
@jigold

jigold Jan 16, 2019

Collaborator

Need to be BitXOr (typos)

assertEvalsTo(ApplyBinaryPrimOp(LogicalRightShift(), i32na, i32na), null)

assertEvalsTo(ApplyBinaryPrimOp(LogicalRightShift(), I64(0xffff5), I32(2)), 0xffff5L >>> 2)
assertEvalsTo(ApplyBinaryPrimOp(LogicalRightShift(), I64(-5), I64(2)), -5L>>> 2)

This comment has been minimized.

Copy link
@jigold

jigold Jan 16, 2019

Collaborator

spacing

tpoterba added some commits Jan 16, 2019

addressed

@tpoterba

This comment has been minimized.

Copy link
Collaborator Author

commented Jan 17, 2019

ready for another review

@jigold

jigold approved these changes Jan 17, 2019

tpoterba added some commits Jan 18, 2019

@danking danking merged commit 2e4cded into hail-is:master Jan 19, 2019

1 check passed

hail-ci-0-1 successful build
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.