Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alert Server Changes - Implement message ID field #5994

Closed
bernd opened this issue Jun 5, 2019 · 0 comments · Fixed by #6074

Comments

@bernd
Copy link
Member

commented Jun 5, 2019

Description

To efficiently get large number of messages from Elasticsearch, we have to use search-after queries in the future. Pagination or scrolling is not efficient and can lead to high load situations in Elasticsearch.

The usage of search-after queries requires the usage of a tie-breaker field for sorting the results. (see search-after documentation) This tie-breaker field must have a unique value per document in Elasticsearch. The _id field has a unique value per document but it's not recommended to use it as a tie-breaker value, because it's not stored as doc value but in fielddata. That means sorting on _id requires Elasticsearch to load lots of data into memory. To make this efficient it's recommended to use a doc value based field with a unique value.

Since we don't have such a field at the moment, we have to add one now so we can use search-after queries in the future.

Implementation Notes

  • The new message ID field needs a gl2_ prefix to make sure we don't overwrite user specific fields
  • Suggestion is to use gl2_message_id
  • The value should be mapped as "keyword" in Elasticsearch, using doc values
  • Instead of using the _id value for the gl2_message_id value, we plan to use an ULID instead
    • This results in shorter IDs (26 characters for ULID vs 36 for UUID) and thus reduced storage usage
    • They are lexicographically sortable (time based UUIDs are as well, but you have to use the correct variant - which we sometimes do and sometimes don't)
  • We are using ULIDs for as event IDs already, using the following Java implementation: https://github.com/huxi/sulky/tree/master/sulky-ulid

@bernd bernd self-assigned this Jul 3, 2019

@bernd bernd added the feature label Jul 3, 2019

bernd added a commit that referenced this issue Jul 3, 2019
Generate a value for the "gl2_message_id" field for each new message
See Message.FIELD_GL2_MESSAGE_ID documentation and the referenced issue
for more details.

Closes #5994

@bernd bernd added the #S label Jul 4, 2019

danotorrey added a commit that referenced this issue Jul 10, 2019
Generate a value for the "gl2_message_id" field for each new message (#…
…6074)

See Message.FIELD_GL2_MESSAGE_ID documentation and the referenced issue
for more details.

Closes #5994
bernd added a commit that referenced this issue Jul 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.