Skip to content

Tracking: Align Bitmap with segmented index lifecycle #6862

@Xuanwo

Description

@Xuanwo

Summary

Bitmap should support the same segment lifecycle used by vector-style distributed indexing:

  1. create_index_uncommitted(fragment_ids=...) builds a durable, uncommitted physical segment.
  2. merge_existing_index_segments(segments) optionally consolidates Bitmap segments.
  3. commit_existing_index_segments(index_name, column, segments) publishes the final physical segments as one logical Bitmap index.

This work must be additive. Existing merged Bitmap indices and the legacy distributed Bitmap workflow based on merge_index_metadata(index_uuid, "BITMAP") must continue to work.

Compatibility boundary

Do not change the legacy meaning of index_uuid in the existing Bitmap distributed path. Today it is a shared merge directory / job id for part_*_bitmap_page_lookup.lance files, not a physical segment id. The new canonical segment path must use independent physical segment UUIDs.

Tasks

Suggested order

  1. Build canonical Bitmap segments.
  2. Commit and query multiple Bitmap segments.
  3. Merge Bitmap segments.
  4. Wire Bitmap into IndexSegmentBuilder.
  5. Preserve and document the legacy path.
  6. Discuss deprecation and default migration separately.

Non-goals for this tracking issue

  • Do not remove merge_index_metadata(index_uuid, "BITMAP").
  • Do not remove BitmapParameters.shard_id.
  • Do not change ordinary create_scalar_index("col", "BITMAP") into a multi-segment build by default.
  • Do not change Python defaults before a separate deprecation / migration discussion.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestindexesRelated to secondary index implementations

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions