BUG:Incorrect array length for StructArray field "captions", expected 8192 got 5364

[sdbds]

I'm sure version 3.0.1 is fine; this bug only appears after updating to version 6.0 or higher.

Traceback (most recent call last):
  File "D:\musubi-tuner-scripts\qinglong-captions\utils\wdtagger.py", line 1249, in <module>
    main(args)
  File "D:\musubi-tuner-scripts\qinglong-captions\utils\wdtagger.py", line 935, in main
    flush_merge_insert()
  File "D:\musubi-tuner-scripts\qinglong-captions\utils\wdtagger.py", line 870, in flush_merge_insert
    dataset.merge_insert(on="uris").when_matched_update_all().execute(table)
  File "D:\musubi-tuner-scripts\qinglong-captions\.venv\Lib\site-packages\lance\dataset.py", line 269, in execute
    return super(MergeInsertBuilder, self).execute(reader)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: LanceError(IO): External error: Encountered internal error. Please file a bug report at https://github.com/lance-format/lance/issues. Error decoding batch: Invalid user input: Invalid argument error: Incorrect array length for StructArray field "captions", expected 8192 got 5364, C:\a\lance\lance\rust\lance-encoding\src\encodings\logical\struct.rs:394:26, C:\a\lance\lance\rust\lance-encoding\src\decoder.rs:2704:26, C:\a\lance\lance\rust\lance\src\dataset\write\merge_insert.rs:945:46

[dantasse]

Hi @sdbds, can you provide any other info to help us debug? Like what code you've been running and something about the table you're running it on?

[sdbds]

@dantasse

Thanks. Here is the code path and table information that triggered this in my project.

Environment

• OS: Windows
• Python: 3.11.10
• Package: pylance / import lance
• Failing version: lance.__version__ == 6.0.1
• PyArrow: 24.0.0
• The same project code path was working with pylance==3.0.1; the failure started after upgrading to 6.x.

Dataset shape

The affected dataset is a Lance dataset used for image caption/tag metadata. In this run there were about 177396 image rows matching mime LIKE 'image/%'.

Relevant schema fields:

("uris", pa.string())              # update key
("mime", pa.string())
("width", pa.int32())
("height", pa.int32())
("channels", pa.int32())
("depth", pa.int32())
("hash", pa.string())
("size", pa.int64())
("has_audio", pa.bool_())
("duration", pa.int32())
("num_frames", pa.int32())
("frame_rate", pa.float32())
("blob", pa.large_binary())
("captions", pa.list_(pa.string()))
("chunk_offsets", pa.list_(pa.int32()))

The dataset is updated by matching on uris and replacing only the captions column with a small Arrow table containing uris and captions.

Old read/count code that triggered the first error

This was the original code before my workaround:

total_images = len(
    dataset.to_table(
        columns=["mime", "captions"],
        filter=(
            "mime LIKE 'image/%'"
            if args.overwrite
            else "mime LIKE 'image/%' and (captions IS NULL OR array_length(captions) = 0)"
        ),
    )
)

scanner = dataset.scanner(
    columns=["uris", "mime", "captions"],
    filter=(
        "mime LIKE 'image/%'"
        if args.overwrite
        else "mime LIKE 'image/%' and (captions IS NULL OR array_length(captions) = 0)"
    ),
    scan_in_order=True,
    batch_size=args.batch_size,
    batch_readahead=16,
    fragment_readahead=4,
    io_buffer_size=32 * 1024 * 1024,
    late_materialization=True,
)

This failed during dataset.to_table(...), before inference or writing:

pyarrow.lib.ArrowInvalid: External error: LanceError(Arrow): Invalid argument error: Attempt to merge two RecordBatch with different sizes: 8192 != 5364, D:\a\lance\lance\rust\lance\src\dataset\scanner.rs:4519:66

In the --overwrite case the pushed-down filter is only:

mime LIKE 'image/%'

but captions is still selected as an output column.

Write/update code that triggered the second error

After I worked around the to_table() count by scanning batches instead, the same affected dataset then failed during merge_insert:

table = pa.table(
    {
        "uris": pa.array([str(path) for path, _ in results], type=pa.string()),
        "captions": pa.array(
            [caption for _, caption in results],
            type=pa.list_(pa.string()),
        ),
    }
)

dataset.merge_insert(on="uris").when_matched_update_all().execute(table)

Error:

OSError: LanceError(IO): External error: Encountered internal error. Please file a bug report at https://github.com/lance-format/lance/issues. Error decoding batch: Invalid user input: Invalid argument error: Incorrect array length for StructArray field "captions", expected 8192 got 5364, C:\a\lance\lance\rust\lance-encoding\src\encodings\logical\struct.rs:394:26, C:\a\lance\lance\rust\lance-encoding\src\decoder.rs:2704:26, C:\a\lance\lance\rust\lance\src\dataset\write\merge_insert.rs:945:46

What seems important

Both failures involve the same row-count mismatch pattern:

• scanner/materialization path: 8192 != 5364
• merge_insert/decode path: expected 8192 got 5364 for StructArray field captions

So this does not look like only a SQL expression issue. It also happens when merge_insert tries to decode existing data while updating a list<string> column.

Workaround used in my project

I fixed the project by avoiding the failing paths:

1. Do not call dataset.to_table(columns=["mime", "captions"], ...) for counting.
2. Scan only with the scalar mime LIKE 'image/%' filter.
3. Set late_materialization=False.
4. Apply the captions is null or empty check in Python on each batch.
5. Stop using merge_insert for incremental captions updates.
6. Write sidecar .txt caption files during processing, then rebuild the Lance dataset from those sidecar files at the end.

Relevant project commits:

• 70d1224 Fix wdtagger Lance caption scan
• ffe459c Avoid wdtagger Lance merge updates

This avoids the crash, but the underlying issue appears to be in decoding/materializing an existing list<string> field after the upgrade to 6.x.

[dantasse]

Thanks! I'll share this with the team.