-
Notifications
You must be signed in to change notification settings - Fork 419
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AWS Billing] Add fingerprint dimension #6749
Conversation
Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
@@ -85,6 +85,9 @@ | |||
object_type: keyword | |||
object_type_mapping_type: "*" | |||
description: Cost explorer group by key values. | |||
- name: group_by.fingerprint | |||
type: keyword |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we actually need to index the fingerprint field?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, because we need to set the dimension as true.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i was meaning to keep the mapping, but add index: false
and doc_values: false
to save disk space.
micro optimization here for sure, but we want to ensure we are applying best practices (and micro optimizations do add up).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That is interesting, I was not aware of those options:
The index option controls whether field values are indexed. It accepts true or false and defaults to true.
If the fingerprint is not indexed, can it still be used to create the _tsid
? What if there is overlapping in documents? If we don't have the fingerprint value, then we will have to search a document based on all fields used for the fingerprint. Is that a good idea?
doc_values: false
The same problem, what if we want to search a document based on the fingerprint?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My main concern is that for TSDB testing we need all dimensions to be searchable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What if there is overlapping in documents?
How do you mean?
If the fingerprint is not indexed, can it still be used to create the _tsid?
I would guess so, but let's find out.
what if we want to search a document based on the fingerprint?
Why would we ever want to do that? we would have to know the content of the document beforehand in order to generate the hash to search.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why would we ever want to do that?
The fingerprint would be a dimension, which means that if documents overlap we will lose data. In that case we can find those documents based on the set of dimensions, including the fingerprint. That is why I think it is useful to be able to search for it.
/test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
approving with the caveat that we should look for storage optimizations on fingerprint fields, since they are not designed to be searchable by the integration user.
🌐 Coverage report
|
Package aws - 1.45.9 containing this change is available at https://epr.elastic.co/search?package=aws |
What does this PR do?
Add dimension fingerprint for
group_by
field.Currently, the dimensions defined for AWS Billing are not enough as the documents keep being overwritten. The fingerprint field solves this problem.
Checklist
changelog.yml
file.Related issues
Relates to #6293.