Support reindex over CCS

[mjmbischoff]

Description

Given we have registered remotes and are able to cross cluster search/replicate I would like to be able to reindex a remote index on the local cluster.

The normal index from remote requires a host to be set. The proxy endpoint used for the remote doesn't work as that's over transport and index from remote uses the http/REST interface. Using the ES rest requires a user to be created with the correct permission, effectively not using the registered remote.

Doing a local index results in index not found:

# normal search works
POST remote1:example-index/_search

# results in index not found
POST _reindex
{
  "source": {
    "index": "remote1:example-index"
  },
  "dest": {
    "index": "my-new-example-index"
  }
}

{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [remote1:example-index]",
        "resource.type" : "index_or_alias",
        "resource.id" : "remote1:example-index",
        "index_uuid" : "_na_",
        "index" : "remote1:example-index"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [remote1:example-index]",
    "resource.type" : "index_or_alias",
    "resource.id" : "remote1:example-index",
    "index_uuid" : "_na_",
    "index" : "remote1:example-index"
  },
  "status" : 404
}

[mjmbischoff]

As supplied by @DaveCTurner :

$ curl --silent --compressed 'http://localhost:9200/_reindex?error_trace' -H 'Content-type: application/json' --data-binary $'{"dest":{"index":"destindex"},"source":{"index":"cluster_one:testindex"}}' | jq '.error.stack_trace' -cMr
java.lang.IllegalArgumentException: Cross-cluster calls are not supported in this context but remote indices were requested: [cluster_one:testindex]
	at org.elasticsearch.server@8.3.2/org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:330)
	at org.elasticsearch.server@8.3.2/org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:314)
	at org.elasticsearch.server@8.3.2/org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:85)
	at org.elasticsearch.reindex.ReindexValidator.validateAgainstAliases(ReindexValidator.java:141)
	at org.elasticsearch.reindex.ReindexValidator.initialValidation(ReindexValidator.java:61)
	at org.elasticsearch.reindex.TransportReindexAction.validate(TransportReindexAction.java:121)
	at org.elasticsearch.reindex.TransportReindexAction.doExecute(TransportReindexAction.java:98)
	at org.elasticsearch.reindex.TransportReindexAction.doExecute(TransportReindexAction.java:36)
	```

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

AFAICT the only obstacle is this validation step that is trying to verify there's no overlap between source and destination indices, and remote source indices cannot also be destination indices so they can just be skipped here. I did a quick experiment which skips this validation and encountered no other problems:

$ curl --silent --compressed 'http://localhost:9200/_reindex?pretty' -H 'Content-type: application/json' --data-binary $'{"dest":{"index":"destindex"},"source":{"index":"cluster_one:testindex"}}'
{
  "took" : 1053,
  "timed_out" : false,
  "total" : 1,
  "updated" : 0,
  "created" : 1,
  "deleted" : 0,
  "batches" : 1,
  "version_conflicts" : 0,
  "noops" : 0,
  "retries" : {
    "bulk" : 0,
    "search" : 0
  },
  "throttled_millis" : 0,
  "requests_per_second" : -1.0,
  "throttled_until_millis" : 0,
  "failures" : [ ]
}

[felix-lessoer]

I was running into the same restriction.
+1 for getting this in as it feels like the expected behavior for users.

[valeriy42]

+1 on fixing this issue.

I also run into it. AFAICS, the only way to overcome it is reindexing specifying "remote" hostname and authentication data. Having CCS support would be much better.

[mjmbischoff]

Thank you @volodk85 !