fix: reduce catalog round-trips in IcebergDocument.hasNext() to improve result read performance

[kunwp1]

What changes were proposed in this PR?

This PR addresses #4289 by optimizing IcebergDocument.hasNext() to minimize redundant catalog round-trips. By introducing a guard condition, we ensure seekToUsableFile() and its subsequent catalog calls are only triggered when the current record iterator is fully exhausted.

1. If the current file has more records, return true immediately.
2. Only if the current file is exhausted, check usableFileIterator.
3. Only if usableFileIterator is also empty, call seekToUsableFile().

Any related issues, documentation, discussions?

Fix #4289

How was this PR tested?

1. Import and use Untitled workflow (9).json.
2. Use a CSV file containing 1M records.
3. Set storage.iceberg.table.commit.batch-size to 1M (matching the total record count).
4. Compare the performance before fix and after fix. For me it was 2m 45s vs 36s.

Was this PR authored or co-authored using generative AI tooling?

No.

[chenlica]

@Xiao-zhen-Liu @bobbai00 Please review it.

[bobbai00]

Scala side LGTM! Can you also check the python side's iceberg document's corresponding logic, change it and teset it?

[kunwp1]

Scala side LGTM! Can you also check the python side's iceberg document's corresponding logic, change it and teset it?

I checked the python side but seems like we don't have such corresponding logic. Can you confirm?

[bobbai00]

Scala side LGTM! Can you also check the python side's iceberg document's corresponding logic, change it and teset it?

I checked the python side but seems like we don't have such corresponding logic. Can you confirm?

Under this folder: https://github.com/apache/texera/tree/main/amber/src/main/python/core/storage/iceberg

For example:
https://github.com/apache/texera/blob/main/amber/src/main/python/core/storage/iceberg/iceberg_document.py and

[kunwp1]

Scala side LGTM! Can you also check the python side's iceberg document's corresponding logic, change it and teset it?

I checked the python side but seems like we don't have such corresponding logic. Can you confirm?

Under this folder: https://github.com/apache/texera/tree/main/amber/src/main/python/core/storage/iceberg

For example: https://github.com/apache/texera/blob/main/amber/src/main/python/core/storage/iceberg/iceberg_document.py and

I checked those files and they don't have the problematic logic. Seems like the implementation on the python side is different. We don't have to fix the python side.