Introduce 64-bit unsigned long field type #60050
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mayya-sharipova
added
>feature
:Search Foundations/Mapping
|
Pinging @elastic/es-search (:Search/Mapping) |
jimczi
reviewed
Jul 22, 2020
|
@jimczi Thanks, Jim, I will look into this. |
mayya-sharipova
force-pushed
the
unsigned64bits_integer
branch
2 times, most recently
from
8861d6d
to
c276ffd
Compare
This field type supports - indexing of integer values from [0, 18446744073709551615] - precise queries (term, range) - sorting and aggregations is based on conversion of long values to double and can be imprecise for large values. Closes elastic#32434
mayya-sharipova
force-pushed
the
unsigned64bits_integer
branch
from
c276ffd
to
dffd748
Compare
jpountz
reviewed
Jul 23, 2020
libs/x-content/src/main/java/org/elasticsearch/common/xcontent/json/JsonXContentParser.java
Outdated
Show resolved
Hide resolved
...nsigned-long/src/main/java/org/elasticsearch/xpack/unsignedlong/UnsignedLongFieldMapper.java
Outdated
Show resolved
Hide resolved
mayya-sharipova
force-pushed
the
unsigned64bits_integer
branch
from
678b883
to
3b1411b
Compare
mayya-sharipova
force-pushed
the
unsigned64bits_integer
branch
from
3b1411b
to
ada3422
Compare
jimczi
approved these changes
Sep 7, 2020
There was a problem hiding this comment.
The change looks great @mayya-sharipova! I left some comments but nothing major.
server/src/main/java/org/elasticsearch/common/io/stream/StreamOutput.java
Outdated
Show resolved
Hide resolved
...ned-long/src/main/java/org/elasticsearch/xpack/unsignedlong/UnsignedLongScriptDocValues.java
Outdated
Show resolved
Hide resolved
jtibshirani
reviewed
Sep 8, 2020
...nsigned-long/src/main/java/org/elasticsearch/xpack/unsignedlong/UnsignedLongFieldMapper.java
Show resolved
Hide resolved
- Convert UnsignedLongFieldMapper to a parametrized form - Small adjustments in UnsignedLongScriptDocValues
mayya-sharipova
added a commit
to mayya-sharipova/elasticsearch
that referenced
this pull request
Oct 16, 2020
Collapse was not working on unsigned_long field, as collapsing was enabled only on KeywordFieldType and NumberFieldType. This introduces a new method `collapseType` to MappedFieldType, that is checked to decide if collapsing should be enabled. Relates to elastic#60050
mayya-sharipova
added a commit
that referenced
this pull request
Oct 16, 2020
Collapse was not working on unsigned_long field, as collapsing was enabled only on KeywordFieldType and NumberFieldType. This introduces a new method `collapseType` to MappedFieldType, that is checked to decide if collapsing should be enabled. Relates to #60050
mayya-sharipova
added a commit
that referenced
this pull request
Oct 16, 2020
UnsignedLongTests for the range agg was using very specific intervals that double type can not distinguish due to lack of precision: 9.223372036854776000E18 == 9.223372036854775807E18 returns true If we add the corresponding range query test, it will return different number of hits than the range agg, as range query unlike range agg doesn't convert valued to double type, and hence more precise. This patch make broader ranges for the range agg test (so values converted to doubles don't loose precision), and hence corresponding range query will return the same number of hits. Relates to #60050
mayya-sharipova
added a commit
that referenced
this pull request
Oct 16, 2020
UnsignedLongTests for the range agg was using very specific intervals that double type can not distinguish due to lack of precision: 9.223372036854776000E18 == 9.223372036854775807E18 returns true If we add the corresponding range query test, it will return different number of hits than the range agg, as range query unlike range agg doesn't convert valued to double type, and hence more precise. This patch make broader ranges for the range agg test (so values converted to doubles don't loose precision), and hence corresponding range query will return the same number of hits. Relates to #60050
mayya-sharipova
added a commit
to mayya-sharipova/elasticsearch
that referenced
this pull request
Oct 19, 2020
Max and min aggs were producing wrong results for unsigned_long field if field was indexed. If field is indexed for max/min aggs instead of field data, we use values from indexed Points, values of which are derived using method pointReaderIfPossible. Before UnsignedLongFieldType#pointReaderIfPossible was incorrectly producing values, as it failed to shift them back to original values. This patch fixes method pointReaderIfPossible to produce correct original values. Relates to elastic#60050
mayya-sharipova
added a commit
that referenced
this pull request
Oct 19, 2020
Max and min aggs were producing wrong results for unsigned_long field if field was indexed. If field is indexed for max/min aggs instead of field data, we use values from indexed Points, values of which are derived using method pointReaderIfPossible. Before UnsignedLongFieldType#pointReaderIfPossible was incorrectly producing values, as it failed to shift them back to original values. This patch fixes method pointReaderIfPossible to produce correct original values. Relates to #60050
mayya-sharipova
added a commit
that referenced
this pull request
Oct 19, 2020
Max and min aggs were producing wrong results for unsigned_long field if field was indexed. If field is indexed for max/min aggs instead of field data, we use values from indexed Points, values of which are derived using method pointReaderIfPossible. Before UnsignedLongFieldType#pointReaderIfPossible was incorrectly producing values, as it failed to shift them back to original values. This patch fixes method pointReaderIfPossible to produce correct original values. Relates to #60050
mayya-sharipova
added a commit
that referenced
this pull request
Oct 19, 2020
Max and min aggs were producing wrong results for unsigned_long field if field was indexed. If field is indexed for max/min aggs instead of field data, we use values from indexed Points, values of which are derived using method pointReaderIfPossible. Before UnsignedLongFieldType#pointReaderIfPossible was incorrectly producing values, as it failed to shift them back to original values. This patch fixes method pointReaderIfPossible to produce correct original values. Relates to #60050
1 task
pugnascotia
pushed a commit
to pugnascotia/elasticsearch
that referenced
this pull request
Oct 21, 2020
Max and min aggs were producing wrong results for unsigned_long field if field was indexed. If field is indexed for max/min aggs instead of field data, we use values from indexed Points, values of which are derived using method pointReaderIfPossible. Before UnsignedLongFieldType#pointReaderIfPossible was incorrectly producing values, as it failed to shift them back to original values. This patch fixes method pointReaderIfPossible to produce correct original values. Relates to elastic#60050
droberts195
added a commit
to droberts195/elasticsearch
that referenced
this pull request
Oct 22, 2020
Adds support for the unsigned_long type to data frame analytics. This type is handled in the same way as the long type. Values sent to the ML native processes are converted to floats and hence will lose accuracy when outside the range where a float can uniquely represent long values. Relates elastic#60050
droberts195
added a commit
that referenced
this pull request
Oct 22, 2020
Adds support for the unsigned_long type to data frame analytics. This type is handled in the same way as the long type. Values sent to the ML native processes are converted to floats and hence will lose accuracy when outside the range where a float can uniquely represent long values. Relates #60050
This was referenced Oct 30, 2020
Merged
1 task
61 tasks
bpintea
added a commit
that referenced
this pull request
Feb 3, 2022
This introduces the UNSIGNED_LONG type to QL following its availability in ES (#60050). The type is mapped to a BigInteger whose value is checked against the UL bounds. The SQL will now support the type as literal and in the arithmetic functions; the non-arithmetic functions however are unchanged (i.e. they still require a long / int parameter where that is the case). The type is version-gated: for the driver SQL clients (only) the server checks their version and in case this is lower than the one introducing the UL support, it fails the request, for queries, or simply hidden in catalog functions (similar to how UNSUPPORTED is currently treated in the similar case) The JDBC tests are adjusted to read the (bwc) version of the driver they are run against and selectively disable part of the tests accordingly. Closes #63312
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This field type supports
Sort values, term aggs keys, script values, doc values fields,
source fields return long (for values <=2^63-1)
and BigInteger (for values > 2^63-1).
Closes #32434