QL: constant_keyword support #53241

astefan · 2020-03-06T19:39:25Z

This PR adds support for constant_keyword data type following its addition in Elasticsearch with #49713.

Addresses #53016

…o 53016_feature

elasticmachine · 2020-03-06T19:39:27Z

Pinging @elastic/es-search (:Search/SQL)

astefan · 2020-03-06T20:33:07Z

@elasticmachine run elasticsearch-ci/2

matriv

Great work! Left some comments.

matriv · 2020-03-06T21:59:38Z

x-pack/plugin/sql/qa/src/main/resources/constant-keyword.csv-spec

+// To mute tests follow example in file: example.csv-spec
+
+//
+// Tests testing field alias (introduced in ES 6.4)


Leftover comment?

matriv · 2020-03-06T22:04:44Z

...in/java/org/elasticsearch/xpack/ql/execution/search/extractor/AbstractFieldHitExtractor.java

@@ -213,6 +214,7 @@ protected Object unwrapMultiValue(Object values) {
    protected boolean isFromDocValuesOnly(DataType dataType) {
        return dataType == KEYWORD // because of ignore_above.
                    || dataType == DATETIME
+                    || dataType == CONSTANT_KEYWORD // because a non-existent value is considered the constant value itself


Could you explain please, I don't get it.

We cannot extract the value of a constant_keyword from _source because if there is no source for the field, Elasticsearch will still consider the field as having the constant value.
Take a look at the example in our docs. No value at indexing time means level is debug. So, for the second document in the sample, there is no _source for level, but the value is still debug in case there is a query filtering on that field.

matriv · 2020-03-06T22:08:11Z

x-pack/plugin/sql/qa/src/main/java/org/elasticsearch/xpack/sql/qa/jdbc/DataLoader.java

@@ -198,6 +195,9 @@ private static void loadEmpDatasetIntoEs(RestClient client, String index, String
                    }
                    hadLastItem = true;
                    bulk.append('"').append(titles.get(f)).append("\":\"").append(fields.get(f)).append('"');
+                    if (titles.get(f).equals("gender") && extraFields) {
+                        bulk.append(",\"extra_gender\":\"M\"");


Personally I would prefer it to be in tha employees.csv file and have a different value from gender like Male and Female.

Imo, it's a too impactful change to do it like this.

Apologies, I mixed up the feature with an enum keyword that can take a set of predefined values in the mapping. It's ok as is, but maybe check if null is also acceptable.

But maybe having is a Male instead of M can help avoid confusion in the future regarding tests?

matriv · 2020-03-06T22:09:58Z

x-pack/plugin/sql/qa/src/main/resources/constant-keyword.csv-spec

+;
+
+aggWithNullFilter
+SELECT COUNT(*) count FROM test_emp_copy WHERE extra_gender IS NOT NULL;


constant_keyword cannot be null, correct?

Theoretically, I think it can be null... haven't tested this.

I guess any type can be null (missing field in the doc)?

For null case, all the documents in the index, for that field, shouldn't have a value. As soon as one document is indexed with something in it for that field, all other documents will "inherit" that value.

matriv · 2020-03-06T22:10:32Z

x-pack/plugin/sql/qa/src/main/resources/constant-keyword.csv-spec

+;
+
+functionOverAlias
+SELECT BIT_LENGTH(extra_gender) bit FROM test_emp_copy ORDER BY extra_gender LIMIT 1;


If we have 2 values Male/Female that should be changed.

A constant_keyword field has only one value.

matriv · 2020-03-06T22:13:30Z

x-pack/plugin/sql/src/test/java/org/elasticsearch/xpack/sql/type/SqlDataTypeConverterTests.java

@@ -631,6 +632,7 @@ public void testCommonType() {
        assertEquals(BOOLEAN, commonType(BOOLEAN, BOOLEAN));
        assertEquals(NULL, commonType(NULL, NULL));
        assertEquals(INTEGER, commonType(INTEGER, KEYWORD));
+        assertEquals(DOUBLE, commonType(DOUBLE, CONSTANT_KEYWORD));


Numeric and constant_keyword not tasted in DataTypeConverterTests of QL.

costin

Great set of tests. Left some comments on removal of the types in case of conflict since I think it might remove index information leading to incorrect mapping.

costin · 2020-03-08T21:15:19Z

x-pack/plugin/ql/src/main/java/org/elasticsearch/xpack/ql/index/IndexResolver.java

@@ -344,6 +351,11 @@ public static IndexResolution mergedMappings(DataTypeRegistry typeRegistry, Stri
                }
            }

+            // if there are both a keyword and a constant_keyword type for this field, only keep the keyword as a common compatible type
+            if (hasCompatibleKeywords) {
+                types.remove(CONSTANT_KEYWORD.esType());


Removing the type all together might remove useful information.
For example if a field a is mapped as keyword in index X and constant_keyword in index Y, the piece above will find the collision and remove the field from Y resulting in a mapping only in X.
On one hand it does indicate that there's only field a however each field has index information underneath so in this case it would be just for index X. I'm not sure though whether we take that into account or not.
What happens is there's no removal?

This scenario is the one where two indices have the same named field with both keyword and constant_keyword types and their mappings need to be merged. For other scenarios (where the same name is used for two fields with different types) we tell the user that the field cannot be shown (because of the different mapping types), but in this very specific case and since these two field types are so similar I chose to display the field with the "common" data type as keyword.

The controversy for what to do in this specific case came from the scenario described here and is specifically tested here.

costin

LGTM

x-pack/plugin/ql/src/main/java/org/elasticsearch/xpack/ql/type/DataTypeConverter.java

bpintea

LGTM

…o 53016_feature

matriv

LGTM Thank you @astefan!

(cherry picked from commit d6cd4ce)

astefan added 3 commits March 6, 2020 21:29

Support for constant_keyword data type

7540f06

Merge branch 'master' of https://github.com/elastic/elasticsearch int…

2f92a65

…o 53016_feature

Update

23a1a7d

astefan added >feature :Analytics/SQL SQL querying v8.0.0 :Analytics/EQL EQL querying v7.7.0 labels Mar 6, 2020

astefan requested review from costin, matriv and bpintea March 6, 2020 21:11

astefan marked this pull request as ready for review March 6, 2020 21:12

matriv reviewed Mar 6, 2020

View reviewed changes

costin reviewed Mar 8, 2020

View reviewed changes

costin approved these changes Mar 9, 2020

View reviewed changes

bpintea reviewed Mar 9, 2020

View reviewed changes

x-pack/plugin/ql/src/main/java/org/elasticsearch/xpack/ql/type/DataTypeConverter.java Show resolved Hide resolved

bpintea approved these changes Mar 9, 2020

View reviewed changes

astefan added 3 commits March 13, 2020 11:54

Address reviews

c5376c9

Address review comments

abad76c

Merge branch 'master' of https://github.com/elastic/elasticsearch int…

1740adb

…o 53016_feature

astefan requested a review from matriv March 13, 2020 14:37

matriv approved these changes Mar 13, 2020

View reviewed changes

astefan merged commit d6cd4ce into elastic:master Mar 16, 2020

astefan deleted the 53016_feature branch March 16, 2020 13:22

astefan added the backport label Mar 16, 2020

astefan added a commit to astefan/elasticsearch that referenced this pull request Mar 16, 2020

QL: constant_keyword support (elastic#53241)

aaf3bb1

(cherry picked from commit d6cd4ce)

astefan added backport pending and removed backport labels Mar 16, 2020

This was referenced Mar 16, 2020

QL: constant_keyword support (#53241) #53602

Merged

QL: Remove CONSTANT_KEYWORD from LiteralTests.java #53606

Merged

astefan added a commit that referenced this pull request Mar 16, 2020

QL: constant_keyword support (#53241) (#53602)

91ca9c5

(cherry picked from commit d6cd4ce)

astefan removed the backport pending label Mar 16, 2020

bpintea mentioned this pull request Mar 16, 2020

Add support for constant_keyword type elastic/elasticsearch-sql-odbc#222

Closed

codebrain mentioned this pull request Apr 1, 2020

7.7.0 meta ticket elastic/elasticsearch-net#4525

Closed

38 tasks

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

QL: constant_keyword support #53241

QL: constant_keyword support #53241

astefan commented Mar 6, 2020 •

edited

Loading

elasticmachine commented Mar 6, 2020

astefan commented Mar 6, 2020

matriv left a comment

matriv Mar 6, 2020

matriv Mar 6, 2020

astefan Mar 7, 2020

matriv Mar 6, 2020

astefan Mar 7, 2020

matriv Mar 7, 2020

matriv Mar 7, 2020

matriv Mar 6, 2020

astefan Mar 7, 2020

bpintea Mar 9, 2020

astefan Mar 13, 2020

astefan Mar 13, 2020

matriv Mar 6, 2020

astefan Mar 7, 2020

matriv Mar 6, 2020

costin left a comment

costin Mar 8, 2020 •

edited

Loading

astefan Mar 9, 2020

costin left a comment

bpintea left a comment

matriv left a comment

QL: constant_keyword support #53241

QL: constant_keyword support #53241

Conversation

astefan commented Mar 6, 2020 • edited Loading

elasticmachine commented Mar 6, 2020

astefan commented Mar 6, 2020

matriv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

costin left a comment

Choose a reason for hiding this comment

costin Mar 8, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

costin left a comment

Choose a reason for hiding this comment

bpintea left a comment

Choose a reason for hiding this comment

matriv left a comment

Choose a reason for hiding this comment

astefan commented Mar 6, 2020 •

edited

Loading

costin Mar 8, 2020 •

edited

Loading