Skip to content

Add some random prefix to split ids. #4824

Open
@fulmicoton

Description

@fulmicoton
Collaborator

S3 creates partitions based on a prefix of the object keys.
Because we use ULID, splits coming from the same day end up on the same servers.
On the search side, most of the queries tend to target the last few days, and end up hitting the same servers.

For this reason, in some case, we can hit the per-prefix rate limiting.

Activity

added a commit that references this issue on Oct 17, 2024
bb3e77f
added 3 commits that reference this issue on Oct 17, 2024
1b0aa58
d207e72
8c90035
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Participants

      @fulmicoton

      Issue actions

        Add some random prefix to split ids. · Issue #4824 · quickwit-oss/quickwit