feat: support manifest version hint

[jackye1995]

Based on benchmarking result in #5947 (comment)

Currently I have only kept JSON format manifest hint support. The exact format to choose requires some further discussions.

[github-actions]

Code Review for PR #5997: feat: support manifest version hint

Summary

This PR adds a version hint optimization for faster manifest lookup on non-lexically ordered object stores (e.g., S3 Express). The approach uses a JSON hint file + HEAD-based probing instead of full directory listing.

P1 Issues

1. Inefficiency when hint file doesn't exist

In current_manifest_path (rust/lance-table/src/io/commit.rs:46-62), when read_version_hint_and_probe returns None quickly (e.g., new dataset, no hint file), the original resolve_version_from_listing future is cancelled and then re-invoked in the fallback path. This doubles the listing cost for datasets without a hint file.

Consider restructuring to avoid this:

// Start list, check hint, if hint fails continue with existing list
let list_fut = resolve_version_from_listing(object_store, base);
tokio::pin!(list_fut);

tokio::select! {
    biased;
    hint_result = read_version_hint_and_probe(object_store, base) => {
        if let Some(location) = hint_result {
            return Ok(location);
        }
        list_fut.await  // Continue with same future
    }
    list_result = &mut list_fut => list_result
}

2. block_in_place usage in write_version_hint

The pattern at line 152-156 spawns a task then immediately blocks on it:

let handle = tokio::spawn(async move { ... });
if sync_write {
    let _ = tokio::task::block_in_place(|| tokio::runtime::Handle::current().block_on(handle));
}

block_in_place doesn't work on single-threaded runtimes (panics). Consider using .await directly when sync_write is true, or documenting this limitation.

Minor Observations

• The test test_commit_uses_version_hint_on_non_lexical_store is thorough and validates the optimization effectively using throttled stores.
• JSON parsing in parse_version_from_json is simple but adequate since format is controlled.

---

🤖 Generated with Claude Code