[python] Support MAP<K, BLOB> in multimodal BLOB reads - #9428
Conversation
5f77a10 to
411fa03
Compare
411fa03 to
996b460
Compare
996b460 to
30094e9
Compare
| else: | ||
| ranges.append(None) | ||
| inline[index] = raw | ||
| targets.append((target, index)) |
There was a problem hiding this comment.
[P2] Avoid per-cell target bookkeeping for scalar BLOB reads
This sends every non-null scalar BLOB cell through the generic target mapping, retaining two target tuples plus the bodies and ranges lists until all reads complete. In an isolated 300k-descriptor run, peak allocations increased from 41.0 MiB on the base to 94.0 MiB on this head; for inline cells they increased from 26.2 MiB to 70.1 MiB, even though no MAP column was involved. Since read_blobs() can materialize large datasets, this O(rows) metadata regression can add substantial GC pressure or trigger OOMs for existing scalar-only callers. Could we keep the previous contiguous offset/slice path for scalar columns and reserve target mappings for MAP entries, for example by passing the MAP column set into this helper?
|
+1 |
Purpose
MultimodalTable.scan().read_blobs()andstream_blobs()recognize only scalar BLOB fields, soMAP<K, BLOB>columns are rejected as non-BLOB columns.This change recognizes MAP BLOB fields, flattens all map-value descriptors into the same coalesced range read used for scalar BLOBs, and reconstructs row-aligned key-value pairs afterward. Map key order, null maps, empty maps, null values, inline values, and mixed scalar/MAP reads are preserved.
The multimodal documentation and end-to-end tests cover both materialized and streaming MAP BLOB reads.
Tests
pytest pypaimon/tests/multimodal_table_test.py -q(68 passed)flake8for changed Python filespython3.6 -m py_compilefor changed Python filesgit diff --check