Skip to content

[python] Support MAP<K, BLOB> in multimodal BLOB reads - #9428

Merged
JingsongLi merged 4 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/multimodal-table-blob-read
Aug 28, 2026
Merged

[python] Support MAP<K, BLOB> in multimodal BLOB reads#9428
JingsongLi merged 4 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/multimodal-table-blob-read

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Purpose

MultimodalTable.scan().read_blobs() and stream_blobs() recognize only scalar BLOB fields, so MAP<K, BLOB> columns are rejected as non-BLOB columns.

This change recognizes MAP BLOB fields, flattens all map-value descriptors into the same coalesced range read used for scalar BLOBs, and reconstructs row-aligned key-value pairs afterward. Map key order, null maps, empty maps, null values, inline values, and mixed scalar/MAP reads are preserved.

The multimodal documentation and end-to-end tests cover both materialized and streaming MAP BLOB reads.

Tests

  • pytest pypaimon/tests/multimodal_table_test.py -q (68 passed)
  • flake8 for changed Python files
  • python3.6 -m py_compile for changed Python files
  • git diff --check

@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as draft August 27, 2026 10:16
@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the codex/multimodal-table-blob-read branch from 5f77a10 to 411fa03 Compare August 27, 2026 10:25
@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Add table-level multimodal BLOB reads [python] Support MAP<K, BLOB> in multimodal BLOB reads Aug 27, 2026
@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the codex/multimodal-table-blob-read branch from 411fa03 to 996b460 Compare August 27, 2026 10:30
@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the codex/multimodal-table-blob-read branch from 996b460 to 30094e9 Compare August 27, 2026 10:31
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review August 27, 2026 11:52
else:
ranges.append(None)
inline[index] = raw
targets.append((target, index))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Avoid per-cell target bookkeeping for scalar BLOB reads

This sends every non-null scalar BLOB cell through the generic target mapping, retaining two target tuples plus the bodies and ranges lists until all reads complete. In an isolated 300k-descriptor run, peak allocations increased from 41.0 MiB on the base to 94.0 MiB on this head; for inline cells they increased from 26.2 MiB to 70.1 MiB, even though no MAP column was involved. Since read_blobs() can materialize large datasets, this O(rows) metadata regression can add substantial GC pressure or trigger OOMs for existing scalar-only callers. Could we keep the previous contiguous offset/slice path for scalar columns and reserve target mappings for MAP entries, for example by passing the MAP column set into this helper?

@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit d9c861f into apache:master Aug 28, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants