Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

isSNP, etc. in Variant are wrong #2

Closed
cseed opened this issue Oct 29, 2015 · 2 comments
Closed

isSNP, etc. in Variant are wrong #2

cseed opened this issue Oct 29, 2015 · 2 comments
Labels

Comments

@cseed
Copy link
Collaborator

cseed commented Oct 29, 2015

From @cseed on August 26, 2015 13:48

Copied from original issue: cseed#1

@cseed
Copy link
Collaborator Author

cseed commented Oct 29, 2015

From @jbloom22 on August 26, 2015 21:12

I've fixed the variant type Boolean methods, added a variantType method which returns the enumerated type, and written tests for all.

@cseed
Copy link
Collaborator Author

cseed commented Oct 29, 2015

@cseed cseed closed this as completed Oct 29, 2015
jbloom22 referenced this issue in jbloom22/hail Jan 19, 2017
jigold added a commit that referenced this issue Aug 9, 2017
* Reference Genome #2

- Added methods to `Variant` that take a `GenomeReference` object
- Reconfigured VariantSubgen to take a Gen[Contig]
- Added a Gen[Contig] from a GenomeReference object

* fixed export bug and changed contig gen slightly
jbloom22 referenced this issue in jbloom22/hail Oct 31, 2017
* Reference Genome #2

- Added methods to `Variant` that take a `GenomeReference` object
- Reconfigured VariantSubgen to take a Gen[Contig]
- Added a Gen[Contig] from a GenomeReference object

* fixed export bug and changed contig gen slightly
danking added a commit that referenced this issue May 4, 2018
* # This is a combination of 22 commits.
# This is the 1st commit message:

apply resettable context

forgot to fix one use of AutoCloseable

fix

add setup iterator

more sensible method ordering

make TrivialContext Resettable

a few more missing resettablecontexts

address comments

apply resettable context

forgot to fix one use of AutoCloseable

fix

add setup iterator

more sensible method ordering

remove rogue element type type

make TrivialContext Resettable

wip

wip

wip

wip

use safe row in join suite

pull over hailcontext

remove Region.clear(newEnd)

add selectRegionValue

# This is the commit message #2:

convert relational.scala
;

# This is the commit message #3:

scope the extract aggregators constfb call

# This is the commit message #4:

scope interpret

# This is the commit message #5:

typeAfterSelect used by selectRegionValue

# This is the commit message #6:

load matrix

# This is the commit message #7:

imports

# This is the commit message #8:

loadbgen converted

# This is the commit message #9:

convert loadplink

# This is the commit message #10:

convert loadgdb

# This is the commit message #11:

convert loadvcf

# This is the commit message #12:

convert blockmatrix

# This is the commit message #13:

convert filterintervals

# This is the commit message #14:

convert ibd

# This is the commit message #15:

convert a few methods

# This is the commit message #16:

convert split multi

# This is the commit message #17:

convert VEP

# This is the commit message #18:

formatting fix

# This is the commit message #19:

add partitionBy and values

# This is the commit message #20:

fix bug in localkeysort

# This is the commit message #21:

fixup HailContext.readRowsPartition use

# This is the commit message #22:

port balding nichols model

* apply resettable context

forgot to fix one use of AutoCloseable

fix

add setup iterator

more sensible method ordering

make TrivialContext Resettable

a few more missing resettablecontexts

address comments

apply resettable context

forgot to fix one use of AutoCloseable

fix

add setup iterator

more sensible method ordering

remove rogue element type type

make TrivialContext Resettable

wip

wip

wip

wip

use safe row in join suite

pull over hailcontext

remove Region.clear(newEnd)

add selectRegionValue

convert relational.scala
;

scope the extract aggregators constfb call

scope interpret

typeAfterSelect used by selectRegionValue

load matrix

imports

loadbgen converted

convert loadplink

convert loadgdb

convert loadvcf

convert blockmatrix

convert filterintervals

convert ibd

convert a few methods

convert split multi

convert VEP

formatting fix

add partitionBy and values

fix bug in localkeysort

fixup HailContext.readRowsPartition use

port balding nichols model

port over table.scala

couple fixes

convert matrix table

remove necessary use of rdd

variety of fixups

wip

add a clear

* Remove direct Region allocation from FilterColsIR

When regions are off-heap, we can allow the globals to live
in a separate, longer-lived Region that is not cleared until
the whole partition is finished. For now, we pay the
memory cost.

* Use RVDContext in MatrixRead zip

This Region will get cleared by consumers.

I introduced the zip primitive which is a safer way to
zip two RVDs because it does not rely on the user correctly
clearing the regions used by the left and right hand sides
of the zip.

* Control the Regions in LoadGDB

I do not fully understand how LoadGDB is working, but a simple
solution to the use-case is to serialize to arrays of bytes
and parallelize those.

I realize there is a proliferation of `coerce` methods. I plan
to trim this down once we do not have RDD and ContextRDD coexisting

* wip

* unify RVD.run

* reset in write

* fixes

* use context region when allocating

* also read RVDs using RVDContext

* formatting

* address comments

* remove unused val

* abstract over boundary

* little fixes

* whoops forgot to clear before persisting

This fixes the LDPrune if you dont clear the region things go wrong.
Not sure what causes that bug. Maybe its something about encoders?

* serialize for shuffles, region.scoped in matrixmapglobals, fix joins

* clear more!

* wip

* wip

* rework GeneralRDD to ease ContextRDD transition

* formatting

* final fixes

* formatting

* merge failures

* more bad merge stuff

* formatting

* remove unnecessary stuff

* remove fixme

* boom!

* variety of merge mistakes

* fix destabilize bug

* add missing newline

* remember to clear the producer region in localkeysort

* switch def to val

* cleanup filteralleles and exporbidbimfam

* fix clearing and serialization issue

* fix BitPackedVectorView

Previously it always assumed the variant struct started at offset
zero, which is not true

* address comments, remove a comment

* remove direct use of Region

* oops

* werrrks, mebbe

* needs cleanup

* fix filter intervals

* fixes

* fixes

* fix filterintervals

* remove unnecessary copy in TableJoin

* and finally fix the last test

* re-use existing CodecSpec definition

* remove unnecessary boundaries

* use RVD abstraction when possible

* formatting

* bugfix: RegionValue must know its region

* remove unnecessary val and comment

* remove unused methods

* eliminate unused constructors

* undo debug change

* formatting

* remove unused imports

* fix bug in tablejoin

* fix RichRDDSuite test

If you have no data, then you have no partitions, not 1 partition
konradjk referenced this issue in konradjk/hail Jun 12, 2018
* # This is a combination of 22 commits.
# This is the 1st commit message:

apply resettable context

forgot to fix one use of AutoCloseable

fix

add setup iterator

more sensible method ordering

make TrivialContext Resettable

a few more missing resettablecontexts

address comments

apply resettable context

forgot to fix one use of AutoCloseable

fix

add setup iterator

more sensible method ordering

remove rogue element type type

make TrivialContext Resettable

wip

wip

wip

wip

use safe row in join suite

pull over hailcontext

remove Region.clear(newEnd)

add selectRegionValue

# This is the commit message #2:

convert relational.scala
;

# This is the commit message #3:

scope the extract aggregators constfb call

# This is the commit message hail-is#4:

scope interpret

# This is the commit message hail-is#5:

typeAfterSelect used by selectRegionValue

# This is the commit message hail-is#6:

load matrix

# This is the commit message hail-is#7:

imports

# This is the commit message hail-is#8:

loadbgen converted

# This is the commit message hail-is#9:

convert loadplink

# This is the commit message hail-is#10:

convert loadgdb

# This is the commit message hail-is#11:

convert loadvcf

# This is the commit message hail-is#12:

convert blockmatrix

# This is the commit message hail-is#13:

convert filterintervals

# This is the commit message hail-is#14:

convert ibd

# This is the commit message hail-is#15:

convert a few methods

# This is the commit message hail-is#16:

convert split multi

# This is the commit message hail-is#17:

convert VEP

# This is the commit message hail-is#18:

formatting fix

# This is the commit message hail-is#19:

add partitionBy and values

# This is the commit message hail-is#20:

fix bug in localkeysort

# This is the commit message hail-is#21:

fixup HailContext.readRowsPartition use

# This is the commit message hail-is#22:

port balding nichols model

* apply resettable context

forgot to fix one use of AutoCloseable

fix

add setup iterator

more sensible method ordering

make TrivialContext Resettable

a few more missing resettablecontexts

address comments

apply resettable context

forgot to fix one use of AutoCloseable

fix

add setup iterator

more sensible method ordering

remove rogue element type type

make TrivialContext Resettable

wip

wip

wip

wip

use safe row in join suite

pull over hailcontext

remove Region.clear(newEnd)

add selectRegionValue

convert relational.scala
;

scope the extract aggregators constfb call

scope interpret

typeAfterSelect used by selectRegionValue

load matrix

imports

loadbgen converted

convert loadplink

convert loadgdb

convert loadvcf

convert blockmatrix

convert filterintervals

convert ibd

convert a few methods

convert split multi

convert VEP

formatting fix

add partitionBy and values

fix bug in localkeysort

fixup HailContext.readRowsPartition use

port balding nichols model

port over table.scala

couple fixes

convert matrix table

remove necessary use of rdd

variety of fixups

wip

add a clear

* Remove direct Region allocation from FilterColsIR

When regions are off-heap, we can allow the globals to live
in a separate, longer-lived Region that is not cleared until
the whole partition is finished. For now, we pay the
memory cost.

* Use RVDContext in MatrixRead zip

This Region will get cleared by consumers.

I introduced the zip primitive which is a safer way to
zip two RVDs because it does not rely on the user correctly
clearing the regions used by the left and right hand sides
of the zip.

* Control the Regions in LoadGDB

I do not fully understand how LoadGDB is working, but a simple
solution to the use-case is to serialize to arrays of bytes
and parallelize those.

I realize there is a proliferation of `coerce` methods. I plan
to trim this down once we do not have RDD and ContextRDD coexisting

* wip

* unify RVD.run

* reset in write

* fixes

* use context region when allocating

* also read RVDs using RVDContext

* formatting

* address comments

* remove unused val

* abstract over boundary

* little fixes

* whoops forgot to clear before persisting

This fixes the LDPrune if you dont clear the region things go wrong.
Not sure what causes that bug. Maybe its something about encoders?

* serialize for shuffles, region.scoped in matrixmapglobals, fix joins

* clear more!

* wip

* wip

* rework GeneralRDD to ease ContextRDD transition

* formatting

* final fixes

* formatting

* merge failures

* more bad merge stuff

* formatting

* remove unnecessary stuff

* remove fixme

* boom!

* variety of merge mistakes

* fix destabilize bug

* add missing newline

* remember to clear the producer region in localkeysort

* switch def to val

* cleanup filteralleles and exporbidbimfam

* fix clearing and serialization issue

* fix BitPackedVectorView

Previously it always assumed the variant struct started at offset
zero, which is not true

* address comments, remove a comment

* remove direct use of Region

* oops

* werrrks, mebbe

* needs cleanup

* fix filter intervals

* fixes

* fixes

* fix filterintervals

* remove unnecessary copy in TableJoin

* and finally fix the last test

* re-use existing CodecSpec definition

* remove unnecessary boundaries

* use RVD abstraction when possible

* formatting

* bugfix: RegionValue must know its region

* remove unnecessary val and comment

* remove unused methods

* eliminate unused constructors

* undo debug change

* formatting

* remove unused imports

* fix bug in tablejoin

* fix RichRDDSuite test

If you have no data, then you have no partitions, not 1 partition
jackgoldsmith4 pushed a commit to jackgoldsmith4/hail that referenced this issue Jun 25, 2018
* # This is a combination of 22 commits.
# This is the 1st commit message:

apply resettable context

forgot to fix one use of AutoCloseable

fix

add setup iterator

more sensible method ordering

make TrivialContext Resettable

a few more missing resettablecontexts

address comments

apply resettable context

forgot to fix one use of AutoCloseable

fix

add setup iterator

more sensible method ordering

remove rogue element type type

make TrivialContext Resettable

wip

wip

wip

wip

use safe row in join suite

pull over hailcontext

remove Region.clear(newEnd)

add selectRegionValue

# This is the commit message hail-is#2:

convert relational.scala
;

# This is the commit message hail-is#3:

scope the extract aggregators constfb call

# This is the commit message hail-is#4:

scope interpret

# This is the commit message hail-is#5:

typeAfterSelect used by selectRegionValue

# This is the commit message hail-is#6:

load matrix

# This is the commit message hail-is#7:

imports

# This is the commit message hail-is#8:

loadbgen converted

# This is the commit message hail-is#9:

convert loadplink

# This is the commit message hail-is#10:

convert loadgdb

# This is the commit message hail-is#11:

convert loadvcf

# This is the commit message hail-is#12:

convert blockmatrix

# This is the commit message hail-is#13:

convert filterintervals

# This is the commit message hail-is#14:

convert ibd

# This is the commit message hail-is#15:

convert a few methods

# This is the commit message hail-is#16:

convert split multi

# This is the commit message hail-is#17:

convert VEP

# This is the commit message hail-is#18:

formatting fix

# This is the commit message hail-is#19:

add partitionBy and values

# This is the commit message hail-is#20:

fix bug in localkeysort

# This is the commit message hail-is#21:

fixup HailContext.readRowsPartition use

# This is the commit message hail-is#22:

port balding nichols model

* apply resettable context

forgot to fix one use of AutoCloseable

fix

add setup iterator

more sensible method ordering

make TrivialContext Resettable

a few more missing resettablecontexts

address comments

apply resettable context

forgot to fix one use of AutoCloseable

fix

add setup iterator

more sensible method ordering

remove rogue element type type

make TrivialContext Resettable

wip

wip

wip

wip

use safe row in join suite

pull over hailcontext

remove Region.clear(newEnd)

add selectRegionValue

convert relational.scala
;

scope the extract aggregators constfb call

scope interpret

typeAfterSelect used by selectRegionValue

load matrix

imports

loadbgen converted

convert loadplink

convert loadgdb

convert loadvcf

convert blockmatrix

convert filterintervals

convert ibd

convert a few methods

convert split multi

convert VEP

formatting fix

add partitionBy and values

fix bug in localkeysort

fixup HailContext.readRowsPartition use

port balding nichols model

port over table.scala

couple fixes

convert matrix table

remove necessary use of rdd

variety of fixups

wip

add a clear

* Remove direct Region allocation from FilterColsIR

When regions are off-heap, we can allow the globals to live
in a separate, longer-lived Region that is not cleared until
the whole partition is finished. For now, we pay the
memory cost.

* Use RVDContext in MatrixRead zip

This Region will get cleared by consumers.

I introduced the zip primitive which is a safer way to
zip two RVDs because it does not rely on the user correctly
clearing the regions used by the left and right hand sides
of the zip.

* Control the Regions in LoadGDB

I do not fully understand how LoadGDB is working, but a simple
solution to the use-case is to serialize to arrays of bytes
and parallelize those.

I realize there is a proliferation of `coerce` methods. I plan
to trim this down once we do not have RDD and ContextRDD coexisting

* wip

* unify RVD.run

* reset in write

* fixes

* use context region when allocating

* also read RVDs using RVDContext

* formatting

* address comments

* remove unused val

* abstract over boundary

* little fixes

* whoops forgot to clear before persisting

This fixes the LDPrune if you dont clear the region things go wrong.
Not sure what causes that bug. Maybe its something about encoders?

* serialize for shuffles, region.scoped in matrixmapglobals, fix joins

* clear more!

* wip

* wip

* rework GeneralRDD to ease ContextRDD transition

* formatting

* final fixes

* formatting

* merge failures

* more bad merge stuff

* formatting

* remove unnecessary stuff

* remove fixme

* boom!

* variety of merge mistakes

* fix destabilize bug

* add missing newline

* remember to clear the producer region in localkeysort

* switch def to val

* cleanup filteralleles and exporbidbimfam

* fix clearing and serialization issue

* fix BitPackedVectorView

Previously it always assumed the variant struct started at offset
zero, which is not true

* address comments, remove a comment

* remove direct use of Region

* oops

* werrrks, mebbe

* needs cleanup

* fix filter intervals

* fixes

* fixes

* fix filterintervals

* remove unnecessary copy in TableJoin

* and finally fix the last test

* re-use existing CodecSpec definition

* remove unnecessary boundaries

* use RVD abstraction when possible

* formatting

* bugfix: RegionValue must know its region

* remove unnecessary val and comment

* remove unused methods

* eliminate unused constructors

* undo debug change

* formatting

* remove unused imports

* fix bug in tablejoin

* fix RichRDDSuite test

If you have no data, then you have no partitions, not 1 partition
danking pushed a commit that referenced this issue Jan 30, 2019
* Prototype for batch Python interface

* wip

* Attempt #2

* change how temp dir is formatted

* change directory structure

* Add tests and actually execute pipeline

* added project

* better error msg

* added docker support

* Add infrastructure for testing, conda, etc.

* support for new pipeline env

* comment out docker test
@akotlar akotlar mentioned this issue Sep 6, 2019
danking pushed a commit that referenced this issue Oct 22, 2020
* [batch] add close billing project functionality

* address comments

* rewrite subquery to take advantage of time_completed index

* fix check

* undo accidental change to query

* tweak table format

* add 'deleted' state for billing projects

* oops

* update estimated-current

* fix

* define custom errors

* address comments

* fix check

* address comments

* fix check

* fix check #2

* fix check #3

* fix check-sql
danking pushed a commit that referenced this issue Sep 27, 2021
* [batch] replace GCS with GoogleStorageAsyncFS in LogStore

* delint

* addr comments

* fix s3 exception handling

* fix catching nonexistent blob in azurefs

* flip version number of instance
iris-garden pushed a commit to iris-garden/hail that referenced this issue Nov 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants