feat(index): consolidate bitmap segments and unindexed data on optimize

[zhangyue19921010]

closes: #7198

[codecov]

Codecov Report

❌ Patch coverage is 92.53438% with 38 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/index/append.rs	96.36%	4 Missing and 8 partials ⚠️
rust/lance-index/src/scalar/bitmap.rs	57.14%	7 Missing and 2 partials ⚠️
rust/lance/src/index/scalar/bitmap.rs	81.39%	3 Missing and 5 partials ⚠️
rust/lance-index/src/scalar/btree.rs	40.00%	6 Missing ⚠️
rust/lance-index/src/scalar.rs	80.00%	1 Missing ⚠️
rust/lance/src/index/create.rs	98.91%	0 Missing and 1 partial ⚠️
rust/lance/src/index/scalar/btree.rs	80.00%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[zhangyue19921010]

Hi @Xuanwo Really appreciate it if you can take a look. Thanks a lot !

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[zhangyue19921010]

move to #7389