Support cross clusters query in ESQL #101640
Conversation
|
Relates #102954 |
dnhatn
force-pushed
the
cross-clusters-esql
branch
4 times, most recently
from
f06d647
to
bf0a972
Compare
dnhatn
force-pushed
the
cross-clusters-esql
branch
3 times, most recently
from
64b78ed
to
00f6d92
Compare
dnhatn
force-pushed
the
cross-clusters-esql
branch
from
00f6d92
to
51ea3c3
Compare
|
@ChrisHegarty, I'm adding you as the reviewer since the new cluster-compute-action is similar to the data-node-compute action, which you are familiar with. |
|
Hi @dnhatn, I've created a changelog YAML for you. |
elasticsearchmachine
added
the
Team:QL (Deprecated)
|
Pinging @elastic/es-ql (Team:QL) |
|
Pinging @elastic/elasticsearch-esql (:Query Languages/ES|QL) |
|
@mark-vieira Thanks for looking. I've switched to the new test framework. I think the new test framework doesn't support multi-clusters with different versions. I work around this limitation by starting two clusters with the old version, then performing a full cluster restart once before running tests. |
|
@nik9000 I've addressed your comments. Can you take another look please? Thank you! |
This shouldn't be necessary. Can you elaborate on what issue you encountered when you attempted to configure one cluster using the BWC version and the other using the current version? |
|
@mark-vieira I checked the nodes of both clusters, they work properly without the workaround. I have removed the workaround in e19bf39. We should be all good here. Thanks Mark. |
...ulti-clusters/src/javaRestTest/java/org/elasticsearch/xpack/esql/ccq/MultiClusterSpecIT.java
Show resolved
Hide resolved
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/ClusterComputeRequest.java
Outdated
Show resolved
Hide resolved
...a/server/multi-clusters/src/javaRestTest/java/org/elasticsearch/xpack/esql/ccq/Clusters.java
Show resolved
Hide resolved
...a/server/multi-clusters/src/javaRestTest/java/org/elasticsearch/xpack/esql/ccq/Clusters.java
Show resolved
Hide resolved
|
@mark-vieira I think I have addressed your comments. Can you please take another look? Thank you! |
|
@nik9000 @mark-vieira Thank you for the reviews. |
This pull request introduces support for cross-clusters query (CCQ) in ESQL. The enrich in CCQ will be implemented in a separate pull request due to its complex semantic nature. The primary change occurs in the ComputeService class, where a cluster-compute-action is introduced. The implementation in this PR is equivalent to CCS with ccs_minimize_round_trips enabled. Currently, our plan is to support a single mode in CCQ. At present, the coordinator on the remote cluster collects pages from data nodes in the same cluster and provides them to the coordinator of the main cluster. This is achieved using two exchange buffers, although a single exchange buffer could suffice. However, the use of two buffers allows for future execution of a plan on this coordinator to perform partial reduce operations, such as limit, topN, and partial-to-partial aggregation. Security and backward compatibility tests have been added in the multi-cluster-search-security and multi-clusters QA modules, respectively.
This pull request introduces support for cross-clusters query (CCQ) in ESQL. The enrich in CCQ will be implemented in a separate pull request due to its complex semantic nature. The primary change occurs in the ComputeService class, where a cluster-compute-action is introduced.
The implementation in this PR is equivalent to CCS with
ccs_minimize_round_tripsenabled. Currently, our plan is to support a single mode in CCQ. At present, the coordinator on the remote cluster collects pages from data nodes in the same cluster and provides them to the coordinator of the main cluster. This is achieved using two exchange buffers, although a single exchange buffer could suffice. However, the use of two buffers allows for future execution of a plan on this coordinator to perform partial reduce operations, such as limit, topN, and partial-to-partial aggregation.Security and backward compatibility tests have been added in the multi-cluster-search-security and multi-clusters QA modules, respectively.