fix: use physical scan stream for update

[wojiaodoubao]

Update rewrites affected rows using the dataset physical schema. Avoid wrapping the scan output in DatasetRecordBatchStream, which may convert internal JSON columns from lance.json/LargeBinary to arrow.json/Utf8 for user-facing reads and cause schema mismatches during rewrite. Add coverage for updating both regular and JSON columns.

Error msg

Traceback (most recent call last):
  File "/Users/bytedance/Project/emr/jinglun-lance-hello-python/main.py", line 298, in <module>
    chunsheng_debug()
    ~~~~~~~~~~~~~~~^^
  File "/Users/bytedance/Project/emr/jinglun-lance-hello-python/main.py", line 291, in chunsheng_debug
    ds.update({'speaker_id': '"SPEAKER_9172"'}, where="name='沈逸'")
    ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bytedance/miniconda3/lib/python3.13/site-packages/lance/dataset.py", line 2577, in update
    return self._ds.update(updates, where, conflict_retries, retry_timeout)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: Encountered internal error. Please file a bug report at https://github.com/lance-format/lance/issues. Expected schema Schema { fields: [Field { name: "id", data_type: Int64, nullable: true }, Field { name: "users_id", data_type: Int64, nullable: true }, Field { name: "user_id", data_type: Utf8, nullable: true }, Field { name: "name", data_type: Utf8, nullable: true }, Field { name: "files", data_type: LargeBinary, nullable: true, metadata: {"ARROW:extension:metadata": "", "ARROW:extension:name": "lance.json"} }, Field { name: "user_tags", data_type: LargeBinary, nullable: true, metadata: {"ARROW:extension:name": "lance.json", "ARROW:extension:metadata": ""} }, Field { name: "output_text", data_type: Utf8, nullable: true }, Field { name: "ai_tags", data_type: LargeBinary, nullable: true, metadata: {"ARROW:extension:name": "lance.json", "ARROW:extension:metadata": ""} }, Field { name: "created_time", data_type: Timestamp(Microsecond, Some("Asia/Shanghai")), nullable: true }, Field { name: "updated_time", data_type: Timestamp(Microsecond, Some("Asia/Shanghai")), nullable: true }, Field { name: "del_flag", data_type: Int32, nullable: true }, Field { name: "speaker_id", data_type: Utf8, nullable: true }], metadata: {} } but got Schema { fields: [Field { name: "id", data_type: Int64, nullable: true }, Field { name: "users_id", data_type: Int64, nullable: true }, Field { name: "user_id", data_type: Utf8, nullable: true }, Field { name: "name", data_type: Utf8, nullable: true }, Field { name: "files", data_type: Utf8, nullable: true, metadata: {"ARROW:extension:metadata": "", "ARROW:extension:name": "arrow.json"} }, Field { name: "user_tags", data_type: Utf8, nullable: true, metadata: {"ARROW:extension:name": "arrow.json", "ARROW:extension:metadata": ""} }, Field { name: "output_text", data_type: Utf8, nullable: true }, Field { name: "ai_tags", data_type: Utf8, nullable: true, metadata: {"ARROW:extension:name": "arrow.json", "ARROW:extension:metadata": ""} }, Field { name: "created_time", data_type: Timestamp(Microsecond, Some("Asia/Shanghai")), nullable: true }, Field { name: "updated_time", data_type: Timestamp(Microsecond, Some("Asia/Shanghai")), nullable: true }, Field { name: "del_flag", data_type: Int32, nullable: true }, Field { name: "speaker_id", data_type: Utf8, nullable: true }], metadata: {} }, /Users/runner/work/lance/lance/rust/lance/src/dataset/write/update.rs:274:24

Process finished with exit code 1

Closes #6329

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[wojiaodoubao]

I ran into error when update a table with json type. Hi @Xuanwo @majin1102 , could you help review this when you have time, thanks very much.

[codecov]

Codecov Report

❌ Patch coverage is 98.57143% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/write/update.rs	98.41%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[wjones127]

Does this solve #6329?

[wjones127]

Looks good! Thank you!