[hail] Add table and matrix table indices #6266

chrisvittal · 2019-06-05T19:38:57Z

This currently is exposed in read_table and read_matrix_table as
undocumented optional parameters _intervals and _filter_intervals
which takes a list of python Intervals that are used either as
a filter or a repartition.

This adds an IndexedRVDSpec as the primary container format for
indexed data, and increments the file version to 1.1.0.

One index file is written per partition. For matrix tables, the offset
to a particular key for the entries is stored in the entries_offset
field of the annotation that an index may contain. We use the new
IndexSpec to retrive the appropriate offset from the index so that we
can seek to the proper offset in the partition.

When writing data with a blocked spec (like the default) we use virtual
index offsets similar to tabix. The high 48 bits are used to indicate
the file offset to the start of a block, and the low 16 bits are used to
indicate the offset from the start of the (possibly decompressed) block.

cc @cseed

tpoterba

very cool change.

tpoterba · 2019-06-06T02:35:38Z

hail/python/hail/ir/matrix_reader.py

+                      filter_intervals=bool)
+    def __init__(self, path, intervals, filter_intervals):
+        if intervals is not None:
+            t = hl.expr.impute_type(intervals)


I think this logic should be in read_matrix_table -- we generally assume that parameters on the IR have been validated.

also, shouldn't there be a check that the interval type agrees with the key type?

Can't do that until this is wrapped in a Table/MatrixTable (python classes). We have no access to the key type at this point. I can certainly move much of the logic into read(_matrix)_table.

Yeah, realized that after I commented and was looking at some of the code i master. I think it would be preferable to do the type lookup in read_matrix_table and then pass it into the IR node, but that's definitely out of scope of this change.

hail/python/hail/ir/matrix_reader.py

hail/python/hail/ir/table_reader.py

hail/python/hail/methods/impex.py

hail/src/main/scala/is/hail/HailContext.scala

hail/src/main/scala/is/hail/expr/ir/MatrixValue.scala

hail/src/main/scala/is/hail/table/Table.scala

hail/src/main/scala/is/hail/variant/MatrixTable.scala

chrisvittal · 2019-06-20T22:09:12Z

I ran the python tests with my latest changes and they all passed. I'll work on addressing comments and adding tests, but this should be ready for another round of code review.

Addressed most of the comments.

chrisvittal · 2019-06-25T19:33:20Z

Okay. This is 'done' (pending more review and revisions).

tpoterba · 2019-06-26T14:19:26Z

I'll try to give this a once-over this afternoon.

hail/src/main/scala/is/hail/expr/ir/TableIR.scala

hail/src/main/scala/is/hail/HailContext.scala

hail/src/main/scala/is/hail/expr/ir/MatrixIR.scala

tpoterba · 2019-06-28T19:51:04Z

Also need to see benchmarks.

chrisvittal · 2019-07-01T00:14:12Z

Still need to write benchmarks, but I wrote a new type to hold the options for the indexed reads, please take a look.

chrisvittal · 2019-07-01T20:00:21Z

Ok, benchmarks are pretty bad, these are the range table benchmarks from #6529

No Index

running write_range_table_p1000...
    Mean, Median: 11.79s, 11.18s
running write_range_table_p100...
    Mean, Median: 4.76s, 4.67s
running write_range_table_p10...
    Mean, Median: 3.28s, 3.03s

Index

running write_range_table_p1000...
    Mean, Median: 28.60s, 28.88s
running write_range_table_p100...
    Mean, Median: 10.17s, 10.20s
running write_range_table_p10...
    Mean, Median: 9.52s, 9.44s

cseed · 2019-07-01T20:11:19Z

For where we are, this seems pretty good. Range table is the worst case because the key consists of all the data. For the key, the index should have roughly 2x cost (stores an extra copy of the key, along with metadata like the file offset). What's more, we're using Java values for the keys and annotations, which is also really bad.

A better to do would be have each node (internal or leaf) have a region that stores the keys, and directly write that out instead of using going from Java object to region value to serialization.

tpoterba · 2019-07-01T20:19:07Z

A better to do would be have each node (internal or leaf) have a region that stores the keys, and directly write that out instead of using going from Java object to region value to serialization.

Chris and I discussed this lasat week.

tpoterba · 2019-07-01T20:19:29Z

Can we do a bit of profiling to figure out where it's spending the time?

This currently is exposed in `read_table` and `read_matrix_table` as undocumented optional parameters `_intervals` and `_filter_intervals` which takes a list of python `Interval`s that are used either as a filter or a repartition. This adds an `IndexedRVDSpec` as the primary container format for indexed data, and increments the file version to 1.1.0. One index file is written per partition. For matrix tables, the offset to a particular key for the entries is stored in the `entries_offset` field of the annotation that an index may contain. We use the new `IndexSpec` to retrive the appropriate offset from the index so that we can seek to the proper offset in the partition. When writing data with a blocked spec (like the default) we use virtual index offsets similar to tabix. The high 48 bits are used to indicate the file offset to the start of a block, and the low 16 bits are used to indicate the offset from the start of the (possibly decompressed) block.

* Only one version * Only one AbstractRVDSpec `read` method. * Being indexed is a property of the spec, not the version

chrisvittal · 2019-07-02T17:17:24Z

It is spending it's time in IndexWriter.+=. For this particular example, we more than triple the data we're writing, before we were writing an int32, now we're writing an int32 row, along with a B-Tree with an int32 key and an int64 offset field. So honestly this is pretty good. This was always going to make writes slower.

tpoterba · 2019-07-02T18:54:21Z

It is spending it's time in IndexWriter.+=

Great, this can also be fixed by not using Java objects. That makes me feel good about things.

Think I got everything

tpoterba previously requested changes Jun 6, 2019

View reviewed changes

chrisvittal force-pushed the indicies branch from 3eea286 to 66e29bb Compare June 6, 2019 03:44

chrisvittal force-pushed the indicies branch from e95038b to 9b4ce49 Compare June 25, 2019 19:32

chrisvittal changed the title ~~WIP Add Indices~~ [hail] Add table and matrix table indices Jun 25, 2019

chrisvittal assigned tpoterba Jun 25, 2019

chrisvittal force-pushed the indicies branch from 9b4ce49 to 9ed6902 Compare June 25, 2019 22:13

chrisvittal force-pushed the indicies branch from 2738163 to 9412c0c Compare June 28, 2019 12:43

tpoterba previously requested changes Jun 28, 2019

View reviewed changes

chrisvittal requested a review from tpoterba July 1, 2019 00:13

chrisvittal force-pushed the indicies branch from 6249db3 to d7b062b Compare July 1, 2019 00:41

chrisvittal added 3 commits July 2, 2019 12:25

Incorporate verbal feedback

1f8ddf6

* Only one version * Only one AbstractRVDSpec `read` method. * Being indexed is a property of the spec, not the version

Add NativeReaderOptions to abstract indexed read options

1786b78

chrisvittal force-pushed the indicies branch from d7b062b to 1786b78 Compare July 2, 2019 16:25

chrisvittal and others added 2 commits July 3, 2019 09:59

Merge branch 'master' into indicies

f1f1ee1

Fix errors after merge

2cf177c

tpoterba approved these changes Jul 3, 2019

View reviewed changes

danking merged commit 277ccc7 into hail-is:master Jul 3, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[hail] Add table and matrix table indices #6266

[hail] Add table and matrix table indices #6266

chrisvittal commented Jun 5, 2019 •

edited

Loading

tpoterba left a comment

tpoterba Jun 6, 2019

tpoterba Jun 6, 2019

chrisvittal Jun 6, 2019

tpoterba Jun 6, 2019

chrisvittal commented Jun 20, 2019

chrisvittal commented Jun 25, 2019

tpoterba commented Jun 26, 2019

tpoterba commented Jun 28, 2019

chrisvittal commented Jul 1, 2019

chrisvittal commented Jul 1, 2019

cseed commented Jul 1, 2019

tpoterba commented Jul 1, 2019

tpoterba commented Jul 1, 2019

chrisvittal commented Jul 2, 2019

tpoterba commented Jul 2, 2019

[hail] Add table and matrix table indices #6266

[hail] Add table and matrix table indices #6266

Conversation

chrisvittal commented Jun 5, 2019 • edited Loading

tpoterba left a comment

Choose a reason for hiding this comment

tpoterba Jun 6, 2019

Choose a reason for hiding this comment

tpoterba Jun 6, 2019

Choose a reason for hiding this comment

chrisvittal Jun 6, 2019

Choose a reason for hiding this comment

tpoterba Jun 6, 2019

Choose a reason for hiding this comment

chrisvittal commented Jun 20, 2019

chrisvittal commented Jun 25, 2019

tpoterba commented Jun 26, 2019

tpoterba commented Jun 28, 2019

chrisvittal commented Jul 1, 2019

chrisvittal commented Jul 1, 2019

No Index

Index

cseed commented Jul 1, 2019

tpoterba commented Jul 1, 2019

tpoterba commented Jul 1, 2019

chrisvittal commented Jul 2, 2019

tpoterba commented Jul 2, 2019

chrisvittal commented Jun 5, 2019 •

edited

Loading