A failed after write refresh can prevent advancing the local checkpoint even when the operations were made durable by the translog

[fcofdez]

Today, when an after write refresh fails, we consider the AsyncAfterWriteAction as failed:

elasticsearch/server/src/main/java/org/elasticsearch/action/support/replication/TransportWriteAction.java

Lines 466 to 471 in 7f83189

	// TODO: Temporary until we fail unpromotable shard
	if (refreshFailure.get() != null) {
	respond.onFailure(refreshFailure.get());
	} else {
	respond.onSuccess(refreshed.get());
	}

This prevents from advancing the local checkpoints with the safely persisted seq nos:

elasticsearch/server/src/main/java/org/elasticsearch/action/support/replication/ReplicationOperation.java

Lines 174 to 195 in 4076aa7

	primaryResult.runPostReplicationActions(new ActionListener<>() {
	@Override
	public void onResponse(Void aVoid) {
	successfulShards.incrementAndGet();
	updateCheckPoints(
	primary.routingEntry(),
	primary::localCheckpoint,
	primary::globalCheckpoint,
	() -> decPendingAndFinishIfNeeded()
	);
	}
	@Override
	public void onFailure(Exception e) {
	logger.trace("[{}] op [{}] post replication actions failed for [{}]", primary.routingEntry().shardId(), opType, request);
	// TODO: fail shard? This will otherwise have the local / global checkpoint info lagging, or possibly have replicas
	// go out of sync with the primary
	finishAsFailed(e);
	}
	});
	}

We should reconsider this behaviour and maybe advance the local checkpoints when the refresh failed for an unpromotable shard.

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)