Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing key error #4799

Closed
konradjk opened this issue Nov 18, 2018 · 2 comments · Fixed by #4904
Closed

Missing key error #4799

konradjk opened this issue Nov 18, 2018 · 2 comments · Fixed by #4904
Assignees

Comments

@konradjk
Copy link
Collaborator

@konradjk konradjk commented Nov 18, 2018

I wonder if related to #4754?

ht = hl.experimental.import_gtf('gs://konradk/gencode.v19.annotation.gtf.bgz', 'GRCh37', True, min_partitions=12)
ht = ht.annotate(gene_id=ht.gene_id.split('\\.')[0],
                 transcript_id=ht.transcript_id.split('\\.')[0],
                 length=ht.interval.end.position - ht.interval.start.position + 1)
coding_regions = ht.filter(ht.feature == 'CDS').select('gene_id', 'transcript_id', 'transcript_type', 'length', 'level')
transcripts = coding_regions.group_by('transcript_id', 'transcript_type', 'gene_id',
                                      transcript_level=coding_regions.level).aggregate(
    cds_length=hl.agg.sum(coding_regions.length),
    num_coding_exons=hl.agg.count()
).key_by('transcript_id')

Afterwards:

transcripts.count()  # fails with error below
transcripts.persist().count() # succeeds

on current master (d33e2d1)

Py4JJavaError: An error occurred while calling z:is.hail.expr.ir.Interpret.interpretPyIR.
: java.util.NoSuchElementException: key not found: interval
	at scala.collection.MapLike$class.default(MapLike.scala:228)
	at scala.collection.AbstractMap.default(Map.scala:59)
	at scala.collection.MapLike$class.apply(MapLike.scala:141)
	at scala.collection.AbstractMap.apply(Map.scala:59)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.AbstractTraversable.map(Traversable.scala:104)
	at is.hail.rvd.RVDType.<init>(RVDType.scala:23)
	at is.hail.expr.types.TableType.<init>(TableType.scala:16)
	at is.hail.expr.types.TableType.copy(TableType.scala:15)
	at is.hail.expr.ir.TableMapRows.<init>(TableIR.scala:592)
	at is.hail.expr.ir.Simplify$$anonfun$tableRules$1.applyOrElse(Simplify.scala:394)
	at is.hail.expr.ir.Simplify$$anonfun$tableRules$1.applyOrElse(Simplify.scala:251)
	at scala.PartialFunction$Lifted.apply(PartialFunction.scala:223)
	at scala.PartialFunction$Lifted.apply(PartialFunction.scala:219)
@konradjk
Copy link
Collaborator Author

@konradjk konradjk commented Nov 19, 2018

Just got something like this on a completely separate pipeline. This appears to occur after grouping on something and not including a previous key. FWIW works with a4f79a3

cc @jbloom22

@konradjk
Copy link
Collaborator Author

@konradjk konradjk commented Dec 6, 2018

Ok confirmed this was not fixed by #4867 - any ideas?

@tpoterba tpoterba self-assigned this Dec 6, 2018
tpoterba added a commit to tpoterba/hail that referenced this issue Dec 6, 2018
tpoterba added a commit to tpoterba/hail that referenced this issue Dec 6, 2018
@danking danking closed this in #4904 Dec 6, 2018
danking added a commit that referenced this issue Dec 6, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

2 participants