[feature](lance) Add verified IVF_FLAT vector index query coverage

[FANNG1]

What problem does this PR solve?

Part of #66495 (tracked by #66340). Related: #65730.

Problem Summary:

The vector_search() query path is algorithm-agnostic — Lance selects the index automatically — but until now only flat search was actually exercised: the fixture from #65730 never created the vector index its tests claimed to cover (the referenced create_vector_search_index.py was never committed), so Lance silently fell back to flat KNN and the "IVF_PQ" regression proved nothing about index usage.

This PR focuses on IVF_FLAT: it adds verified IVF_FLAT query coverage and repairs the IVF_PQ fixture the existing suite depends on. No FE/BE/SQL/Thrift/lance-c changes — the deliverable is a verified fixture plus regression coverage. The remaining algorithms (IVF_SQ, IVF_HNSW_FLAT, IVF_HNSW_SQ, IVF_HNSW_PQ) follow the same pattern in follow-up PRs.

1. Offline fixture generator

docker/thirdparties/docker-compose/iceberg/scripts/lance_build_preinstalled_catalog.py, pinned to pylance 4.0.1 (matching lance-c v0.1.2 → lance-rs 4.0.1 used by the BE):

• Builds the Directory V2 catalog through the namespace API (never predicting hashed storage paths): re-registers all_types, creates the doris namespace, vector_search_ivf_flat and vector_search_ivf_pq.
• Deterministic data: 1024 rows, two 512-row fragments, 16-dim Float32, embedding[j] = (row_id - 1) + j. For a query equal to row r's vector, the exact squared L2 distance of row n is 16 * (n - r)^2 — an integer ladder, tie-free at the endpoints, exact in Float32.
• Self-check asserts: Lance plans contain ANNSubIndex/ANNIvfPartition; every index covers both fragments; IVF_FLAT full-probe equals flat search exactly; and the row-256 partition-boundary discriminator holds per table.
• The generated catalog (404K) is committed; the manifest is compacted back to one version / one data file / three scalar indexes.

2. Regression suites

• test_lance_vector_search now targets the real vector_search_ivf_pq table; new test_lance_vector_search_ivf_flat suite.
• Silent-fallback discriminator in both suites: querying the row-256 partition boundary with nprobes=1 must differ from the flat result. A pipeline that ignores use_index/nprobes returns exactly the flat rows and fails the assertion — golden files alone cannot catch this.
• IVF_FLAT additionally asserts indexed == flat programmatically at full partition probes (its algorithm guarantee — vectors stored unquantized). IVF_PQ freezes refine_factor-reranked results in goldens without claiming algorithm guarantees.

3. Verified compatibility

The pylance-4.0.1-generated manifest was exercised against lance-java 4.0.0 (the version inside lance-spark-bundle 0.4.0, which registers predicate_pushdown at container startup) and lance-java 9.1.0-beta.3 (the FE Directory Namespace client): list/describe/register/deregister all work, and the pylance self-check passes on the java-mutated manifest.

Recorded fixture behaviors (pinned Lance version): cosine query on an L2 index warns and falls back to flat search (no error); raw PQ distances are quantization approximations, exact after refine_factor reranking.

Draft status

Draft until the .out goldens are generated by a full docker regression run (--forceGenOut), which also provides the FE → BE → lance-c end-to-end evidence. Everything above the Doris query layer is verified offline by the generator self-check.

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:

• Behavior changed:

  • No.
  • Yes.

• Does this need documentation?

  • No.
  • Yes. (compatibility matrix doc to follow in doris-website; behaviors recorded by the generator)

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

https://claude.ai/code/session_014jKR13f5jdqs8NFyEuCnAN

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?