Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Match issue in extract interval filters with a locus argument. #6458

Closed
chrisvittal opened this issue Jun 24, 2019 · 4 comments

Comments

@chrisvittal
Copy link
Collaborator

commented Jun 24, 2019

Script:

#!/usr/bin/env python3
import hail as hl
hl.init(log='/dev/null')
mt = hl.import_vcf('src/test/resources/sample.vcf')
mt.filter_rows(mt.locus < hl.Locus('1', 1)).show()

Output:

2019-06-24 19:12:05 WARN  Utils:66 - Your hostname, wp086-661 resolves to a loopback address: 127.0.1.1; using 10.1.8.50 instead (on interface wlp2s0)
2019-06-24 19:12:05 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2019-06-24 19:12:06 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Running on Apache Spark version 2.4.0
SparkUI available at http://wp086-661.broadinstitute.org:4040
Welcome to
     __  __     <>__
    / /_/ /__  __/ /
   / __  / _ `/ / /
  /_/ /_/\_,_/_/_/   version 0.2.16-e95038bbed35
LOGGING: writing to /dev/null
Traceback (most recent call last):
  File "/tmp/x", line 4, in <module>
    mt.filter_rows(mt.locus < hl.Locus('1', 1)).show()
  File "</home/BROAD.MIT.EDU/cvittal/.cache/hail-env/lib/python3.6/site-packages/decorator.py:decorator-gen-1000>", line 2, in show
  File "/home/BROAD.MIT.EDU/cvittal/src/hail-alt/hail/python/hail/typecheck/check.py", line 585, in wrapper
    return __original_func(*args_, **kwargs_)
  File "/home/BROAD.MIT.EDU/cvittal/src/hail-alt/hail/python/hail/matrixtable.py", line 2569, in show
    actual_n_cols = self.count_cols()
  File "</home/BROAD.MIT.EDU/cvittal/.cache/hail-env/lib/python3.6/site-packages/decorator.py:decorator-gen-994>", line 2, in count_cols
  File "/home/BROAD.MIT.EDU/cvittal/src/hail-alt/hail/python/hail/typecheck/check.py", line 585, in wrapper
    return __original_func(*args_, **kwargs_)
  File "/home/BROAD.MIT.EDU/cvittal/src/hail-alt/hail/python/hail/matrixtable.py", line 2404, in count_cols
    return Env.backend().execute(ir)
  File "/home/BROAD.MIT.EDU/cvittal/src/hail-alt/hail/python/hail/backend/backend.py", line 108, in execute
    result = json.loads(Env.hc()._jhc.backend().executeJSON(self._to_java_ir(ir)))
  File "/home/BROAD.MIT.EDU/cvittal/.local/opt/spark/spark-2.4.0-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
  File "/home/BROAD.MIT.EDU/cvittal/src/hail-alt/hail/python/hail/utils/java.py", line 221, in deco
    'Error summary: %s' % (deepest, full, hail.__version__, deepest)) from None
hail.utils.java.FatalError: MatchError: locus<GRCh37> (of class is.hail.expr.types.virtual.TLocus)

Java stack trace:
scala.MatchError: locus<GRCh37> (of class is.hail.expr.types.virtual.TLocus)
	at is.hail.expr.ir.ExtractIntervalFilters$.minimumValueByType(ExtractIntervalFilters.scala:42)
	at is.hail.expr.ir.ExtractIntervalFilters$.openInterval(ExtractIntervalFilters.scala:94)
	at is.hail.expr.ir.ExtractIntervalFilters$$anonfun$extractAndRewrite$6.apply(ExtractIntervalFilters.scala:205)
	at is.hail.expr.ir.ExtractIntervalFilters$$anonfun$extractAndRewrite$6.apply(ExtractIntervalFilters.scala:201)
	at scala.Option.flatMap(Option.scala:171)
	at is.hail.expr.ir.ExtractIntervalFilters$.extractAndRewrite(ExtractIntervalFilters.scala:201)
	at is.hail.expr.ir.ExtractIntervalFilters$.extractAndRewrite(ExtractIntervalFilters.scala:151)
	at is.hail.expr.ir.ExtractIntervalFilters$.extractPartitionFilters(ExtractIntervalFilters.scala:249)
	at is.hail.expr.ir.ExtractIntervalFilters$$anonfun$apply$2.apply(ExtractIntervalFilters.scala:266)
	at is.hail.expr.ir.ExtractIntervalFilters$$anonfun$apply$2.apply(ExtractIntervalFilters.scala:254)
	at is.hail.expr.ir.RewriteBottomUp$.is$hail$expr$ir$RewriteBottomUp$$rewrite$1(RewriteBottomUp.scala:15)
	at is.hail.expr.ir.RewriteBottomUp$$anonfun$1.apply(RewriteBottomUp.scala:6)
	at is.hail.expr.ir.RewriteBottomUp$$anonfun$1.apply(RewriteBottomUp.scala:6)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.AbstractTraversable.map(Traversable.scala:104)
	at is.hail.expr.ir.RewriteBottomUp$.is$hail$expr$ir$RewriteBottomUp$$rewrite$1(RewriteBottomUp.scala:6)
	at is.hail.expr.ir.RewriteBottomUp$$anonfun$1.apply(RewriteBottomUp.scala:6)
	at is.hail.expr.ir.RewriteBottomUp$$anonfun$1.apply(RewriteBottomUp.scala:6)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.AbstractTraversable.map(Traversable.scala:104)
	at is.hail.expr.ir.RewriteBottomUp$.is$hail$expr$ir$RewriteBottomUp$$rewrite$1(RewriteBottomUp.scala:6)
	at is.hail.expr.ir.RewriteBottomUp$.apply(RewriteBottomUp.scala:25)
	at is.hail.expr.ir.ExtractIntervalFilters$.apply(ExtractIntervalFilters.scala:254)
	at is.hail.expr.ir.Optimize$.optimize(Optimize.scala:19)
	at is.hail.expr.ir.Optimize$.apply(Optimize.scala:49)
	at is.hail.expr.ir.CompileAndEvaluate$$anonfun$optimizeIR$1$1.apply(CompileAndEvaluate.scala:20)
	at is.hail.expr.ir.CompileAndEvaluate$$anonfun$optimizeIR$1$1.apply(CompileAndEvaluate.scala:20)
	at is.hail.utils.ExecutionTimer.time(ExecutionTimer.scala:20)
	at is.hail.expr.ir.CompileAndEvaluate$.optimizeIR$1(CompileAndEvaluate.scala:20)
	at is.hail.expr.ir.CompileAndEvaluate$.apply(CompileAndEvaluate.scala:24)
	at is.hail.backend.Backend.execute(Backend.scala:86)
	at is.hail.backend.Backend.executeJSON(Backend.scala:92)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:748)



Hail version: 0.2.16-e95038bbed35
Error summary: MatchError: locus<GRCh37> (of class is.hail.expr.types.virtual.TLocus)
@tpoterba

This comment has been minimized.

Copy link
Collaborator

commented Jun 24, 2019

This should be a type error. I don't think you should be able to use comparison operators on non-primitive types

@tpoterba

This comment has been minimized.

Copy link
Collaborator

commented Jun 24, 2019

oh, right, someone contributed that change in December...

@tpoterba tpoterba self-assigned this Jun 24, 2019

@chrisvittal

This comment has been minimized.

Copy link
Collaborator Author

commented Jun 24, 2019

Strong disagree, locus1 < locus2 is well formed and valid. We definitely should be able to compare them.

@tpoterba

This comment has been minimized.

Copy link
Collaborator

commented Jun 24, 2019

what about array<set<dict<str, str>>> < array<set<dict<str, str>>>? I agree that Locus comparisons should be specially supported, but I think that the change to expose comparisons on any types was wrong.

tpoterba added a commit to tpoterba/hail that referenced this issue Jun 25, 2019

danking added a commit that referenced this issue Jun 25, 2019

[hail][bugfix] Fix ExtractIntervalFilter comparisons (#6459)
* [hail][bugfix] Fix ExtractIntervalFilter comparisons

Fixes #6458

* make consistent

* Fix spurious rewrite

* bump!

* bump!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.