[Transform] 8.2 transform can not read from < 8.2.0 CCS source

[pheyos]

Affected version: 8.2.0
Fixed with: 8.2.1

After upgrading the ML QA long running cluster from 8.1.3 to 8.2.0, transforms with CCS source <8.2.0 (8.1.3 and 8.1.1 in our case) are failing with

Failed to retrieve checkpointing info for transform [ccs_monitoring_cluster_data_stream_with_ilm_on] org.elasticsearch.xpack.transform.checkpoint.CheckpointException: Failed to create checkpoint
[...]
Caused by: org.elasticsearch.ElasticsearchSecurityException: failed to verify signed authentication information

The remote cluster log shows related entries:

caught exception while trying to read authentication from request [transport request action [indices:monitor/stats]] java.io.EOFException: null
[...]
Authentication using apikey failed - unable to find apikey with id <REDACTED>

Running an _update request on the transforms to update the headers._xpack_security_authentication didn't fix it.

Mitigation

• upgrade the CCS remote to >= 8.2.0

or

• upgrade the cluster to >= 8.2.1

[hendrikmuhs]

Thanks @pheyos

I am able to reproduce this issue. It seems to me like a regression originating from #84473. The 8.1 remote doesn't understand the request and fails.

As CCS supports only one minor back, the problem will disappear once the cluster runs on 8.3 or higher, because the remote must run on 8.2 in this case.

For 8.2 I suggest a workaround as part of 8.2.1 where we go into compatibility mode if we catch the exception above. Transform will automatically retry in this case. As there are no further releases planned for 8.1 this is the best option IMO.

Unfortunately this workaround does not work.

@ywangd Please have a look. Can you think of any other workaround?

Lesson learned: We have CCS compat system tests, however we are testing the same cluster versions, I will try to extent it to additionally test against a remote from the previous minor version. -> #86727

[ywangd]

It turns out to be a long standing bug that just got surfaced. It is a rare code path and also we didn't have any version difference for Authentication object till 8.2.

The following branch of server transport interceptor does not take connection version into consideration. This leads to an Authentication object of v8.2 being sent to a v8.1.3 node which does not know how to decode it.

elasticsearch/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/transport/SecurityServerTransportInterceptor.java

Lines 105 to 117 in 3e7f523

	} else if (AuthorizationUtils.shouldSetUserBasedOnActionOrigin(threadPool.getThreadContext())) {
	AuthorizationUtils.switchUserBasedOnActionOriginAndExecute(
	threadPool.getThreadContext(),
	securityContext,
	(original) -> sendWithUser(
	connection,
	action,
	request,
	options,
	new ContextRestoreResponseHandler<>(threadPool.getThreadContext().wrapRestorable(original), handler),
	sender
	)
	);

I adjusted the labels and self assigned.

[elasticmachine]

Pinging @elastic/es-security (Team:Security)