Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for VERSION field type in SQL and EQL #87590

Conversation

luigidellaquila
Copy link
Contributor

Re-submitting #85502, reverted after a CI failure (error on serialization of ConstantProcessor containing a Version object).

The problem was reported by a randomized test, probably it was not identified during the review phase because it's unlikely to happen on CSV spec test suite, due to relatively small result sets.

The applied fix consists in letting ConstantProcessor manage Version objects as a special case and serialize them directly, without relying on StreamInput/Output generic value serialization.
A more natural solution would have been to let StreamInput/Output manage Version objects natively, but it was discarded because Version is in x-pack and cannot be directly referenced/instantiated in Server module (unless we move it and change its license).

Fixes #83375

luigidellaquila and others added 30 commits March 30, 2022 17:40
…sion_data_type_in_ql' into enhancement/support_version_data_type_in_ql
…sion_data_type_in_ql' into enhancement/support_version_data_type_in_ql
and add new test cases
- better type checks (avoid unpredictable automatic casts)
- support multiple casts
- add further test cases
@elasticmachine elasticmachine added the Team:QL (Deprecated) Meta label for query languages team label Jun 13, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-ql (Team:QL)

@elasticsearchmachine
Copy link
Collaborator

Hi @luigidellaquila, I've created a changelog YAML for you.

Copy link
Contributor

@Luegg Luegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks generally good to me. But can you give an example of the tests that failed?

@@ -0,0 +1,6 @@
pr: 85502
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this file is no longer needed

@luigidellaquila
Copy link
Contributor Author

@Luegg the failing stack trace was the following:

org.elasticsearch.xpack.sql.qa.single_node.JdbcCsvSpecIT > test {version.testCastConstantToVersion2} FAILED
    java.sql.SQLException: Server sent bad type [illegal_argument_exception]. Original type was [can not write type [class org.elasticsearch.xpack.versionfield.Version]]. [java.lang.IllegalArgumentException: can not write type [class org.elasticsearch.xpack.versionfield.Version]
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamOutput.writeGenericValue(StreamOutput.java:828)
        at org.elasticsearch.xpack.ql.expression.gen.processor.ConstantProcessor.writeTo(ConstantProcessor.java:68)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamOutput.writeNamedWriteable(StreamOutput.java:1097)
        at org.elasticsearch.xpack.ql.execution.search.extractor.ComputingExtractor.writeTo(ComputingExtractor.java:56)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamOutput.writeNamedWriteable(StreamOutput.java:1097)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamOutput.writeNamedWriteableList(StreamOutput.java:1203)
        at org.elasticsearch.xpack.sql.execution.search.SearchHitCursor.writeTo(SearchHitCursor.java:80)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamOutput.writeNamedWriteable(StreamOutput.java:1097)
        at org.elasticsearch.xpack.sql.session.Cursors.encodeToString(Cursors.java:82)
        at org.elasticsearch.xpack.sql.session.Cursors.encodeToString(Cursors.java:73)
        at org.elasticsearch.xpack.sql.plugin.TransportSqlQueryAction.createResponse(TransportSqlQueryAction.java:225)
        at org.elasticsearch.xpack.sql.plugin.TransportSqlQueryAction.createResponseWithSchema(TransportSqlQueryAction.java:206)
        at org.elasticsearch.xpack.sql.plugin.TransportSqlQueryAction.lambda$operation$0(TransportSqlQueryAction.java:170)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162)
        at org.elasticsearch.xpack.sql.execution.search.SearchHitCursor.handle(SearchHitCursor.java:181)
        at org.elasticsearch.xpack.sql.execution.search.Querier$SearchHitActionListener.handleResponse(Querier.java:616)
        at org.elasticsearch.xpack.sql.execution.search.Querier$BaseActionListener.onResponse(Querier.java:686)
        at org.elasticsearch.xpack.sql.execution.search.Querier$BaseActionListener.onResponse(Querier.java:665)
        at org.elasticsearch.xpack.sql.execution.search.Querier.lambda$searchWithPointInTime$0(Querier.java:159)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.client.internal.node.NodeClient$ActionResponseTaskListener.onResponse(NodeClient.java:175)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:171)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:165)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.action.ActionListener$RunAfterActionListener.onResponse(ActionListener.java:367)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.sendSearchResponse(AbstractSearchAsyncAction.java:722)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.action.search.FetchLookupFieldsPhase.run(FetchLookupFieldsPhase.java:75)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:469)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:463)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.action.search.ExpandSearchPhase.onPhaseDone(ExpandSearchPhase.java:151)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:105)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:469)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:463)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:269)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$2(FetchSearchPhase.java:106)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:115)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:90)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:768)
        at org.elasticsearch.server@8.4.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)

It reproduced deterministically with a specific seed:

./gradlew ':x-pack:plugin:sql:qa:server:single-node:javaRestTest' --tests "org.elasticsearch.xpack.sql.qa.single_node.JdbcCsvSpecIT" -Dtests.method="test {version.testScriptsOperatorsOrderBy}" -Dtests.seed=8B1768D7955260FA -Dtests.locale=hi -Dtests.timezone=Etc/UCT -Druntime.java=17

Copy link
Contributor

@astefan astefan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@bpintea bpintea left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@costin costin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@luigidellaquila
Copy link
Contributor Author

@elasticmachine update branch

@luigidellaquila
Copy link
Contributor Author

@elasticmachine update branch

@luigidellaquila luigidellaquila merged commit 62f27f5 into elastic:master Jun 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Analytics/EQL EQL querying :Analytics/SQL SQL querying >enhancement Team:QL (Deprecated) Meta label for query languages team v8.4.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

EQL support for fields of type version
7 participants