Skip to content

Segment membership: Bitmap sharding and parallel backfill #7419

@khvn26

Description

@khvn26

Together with Flagsmith/flagsmith-engine#299, this unlocks:

  • read-latency SLO tightening for 100M+ environments
  • backfill SLO bucket tightening for 10M+ environments

Scope

  • Migrate AtomBitmap from OneToOneField to (atom, shard_id) unique. Background job to split existing blobs by ord range.
  • Read path fetches all shards for an atom in one query and unions in memory.
  • process_segment_canonical_changed becomes a fan-out task spawning one child task per shard.

Acceptance criteria

  • Per-shard blob caps at ~128 KB at the 870M-identity scale.
  • Read latency p95 enters the "≤ 10M" SLO envelope even for the 870M env after sharding.
  • Backfill of the 870M env completes in less than 24h on N workers (N TBD by review).

Metadata

Metadata

Assignees

No one assigned

    Labels

    apiIssue related to the REST API

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions