Support pyarrow output for GenomicsDB queries #51
Conversation
| self._genomicsdb.query_variant_calls(processor, array_name, scan_range) | ||
|
|
||
| if non_blocking: | ||
| query_thread = threading.Thread(target=query_calls) |
There was a problem hiding this comment.
Will be changing this to using the thread pool from asyncio
requirements.txt
Outdated
| @@ -28,3 +28,4 @@ | |||
| numpy>=1.24.0 | |||
| pandas>=2.1.0 | |||
| protobuf>=4.21.1 | |||
| pyarrow | |||
There was a problem hiding this comment.
Could a version specifier be added to this dependency?
There was a problem hiding this comment.
@oliviawebber , any suggestions for pyarrow versions? I don't want to introduce incompatibilities with other packages.
There was a problem hiding this comment.
If there's no known minimum, my usual rule of thumb is to pick a version from like ~1 year ago which would give 14.0.0. The one library I know we use that might eventually pull in pyarrow looks like it requires >= 7.0.0.
There was a problem hiding this comment.
Sorta related: I ran into this issue https://stackoverflow.com/questions/78634235/numpy-dtype-size-changed-may-indicate-binary-incompatibility-expected-96-from recently. Based on that (and maybe this migration guide https://numpy.org/devdocs/numpy_2_0_migration_guide.html) do we also need to pin to numpy<2.0.0?
There was a problem hiding this comment.
Yes, I am using numpy C interface that may need porting to 2. So will have
numpy>=1.25,<2.0.0
pyarrow>=14.0.0
src/genomicsdb.pyx
Outdated
| schema_obj = _ArrowSchemaWrapper._import_from_c_capsule(schema_capsule) | ||
| schema = pa.schema(schema_obj.children_schema) | ||
| yield schema.serialize().to_pybytes() | ||
| if arrow_array and arrow_schema: |
There was a problem hiding this comment.
The and arrow_schema part of this check seems redundant given the arrow_schema == NULL check just above, or is there some truthiness conversion?
| break | ||
|
|
||
| if non_blocking: | ||
| query_thread.join() |
There was a problem hiding this comment.
Do we need to clean up the array and schema here? Or does the client do that?
There was a problem hiding this comment.
The client does the cleanup here as the operations are zero copy. See
GenomicsDB-Python/src/utils.pxi
Lines 102 to 103 in 4e37170
GenomicsDB-Python/src/utils.pxi
Lines 105 to 106 in 4e37170
…csDB-Python#51, address review comments and some cleanup
…lls() (#338) * Support arrow output using nanoarrow in respnse to query_variant_calls() * Introduce semaphore for separate array_schema() see GenomicsDB/GenomicsDB-Python#51, address review comments and some cleanup * Test coverage for ArrowVariantCallProcessor::allocate_schema
No description provided.