How to retry when response code is `409`

[horihiro]

Could you please let me know how to retry when an HTTP response code is 409 Conflict with @azure/core-pipeline-rest package?

As far as reading the code, the package seems to use 408, 500, 502-504, 506 or more as response codes that are need to retry.

azure-sdk-for-js/sdk/core/core-rest-pipeline/src/retryStrategies/exponentialRetryStrategy.ts

Lines 81 to 94 in 8299623

	/**
	* A response is a retry response if it has status codes:
	* - 408, or
	* - Greater or equal than 500, except for 501 and 505.
	*/
	export function isExponentialRetryResponse(response?: PipelineResponse): boolean {
	return Boolean(
	response &&
	response.status !== undefined &&
	(response.status >= 500 || response.status === 408) &&
	response.status !== 501 &&
	response.status !== 505
	);
	}

Or please add 409 to the retry target status code.

Background:
We are trying to develop a GitHub actions aca-review-apps that operates revision status of Azure Container Apps and uses @azure/core-pipeline-rest via @azure/arm-appcontainers.
If the operation by the action and the one from Azure portal are conflicted, then Azure platform returns HTTP 409 error as the action result. In this case we think it's needed to retry the operation.

[xirzec]

I'm not very familiar with Container Apps, but I don't think 409 is a retryable response code from a HTTP perspective as it usually indicates there is a problem with the request itself (e.g. you are issuing a PUT to an old version.)

Are you saying that an exact replay of the same request to this service will succeed without any conflict resolution needed?

/cc Jeffrey Richter (@JeffreyRichter) Johan Stenberg (ex-msft) (@johanste)

[JeffreyRichter]

Correct, normally a 409 is not retryable.

[horihiro]

Thank you for your response, Jeff Fisher (@xirzec) Jeffrey Richter (@JeffreyRichter).

In this ACA case, the REST API for update(Container Apps - Update) returns 409 during update from other clients, but after finishing the update from others, the API with exact same request returns 202.
This is why I think 409 in this case is retryable.

[JeffreyRichter]

If the PATCH operation is implemented as a long-running operation (which it sounds like it is), then retrying would potentially take a long time (by definition). For Azure data-plane operations, we forbid implementing PATCH as a long-running operation and instead require it be implemented as a POST-action LRO and if multiple clients issue this simultaneously, then the operations are queued and executed sequentially. So, no 409 would be returned.

I think the container apps should consider changing their pattern to this as the data-plane pattern is better. A 409 should not be considered retryable. Depending on the language, you might be able to get the SDK to retyr on a 409 but you wouldn't want this behavior for any other operation on the client object.

[horihiro]

I understood that:

• it's not appropreate for Azure operations to return 409 to PATCH request to update resources.
• For long-running-operation, the requests should be POST requets, queued, and processed sequentially.

Is it right?

[JeffreyRichter]

First, data-plane and control-plane do not always follow the same guidelines. Second, there are some HTTP standards that should be adhered to and some parts of the standard can be ambiguous.

A 409-Conflict indicates that the current state of the resource and the operation being attempted are in conflict with each other. There is no reason to think that the resource will dynamically change to resolve the conflict. Instead, it is more likely that the client request has to change to resolve the conflict and this is why retrying shouldn't make sense for a 409 and also why the service should not return a 409 in this case.

So this is what is wrong. Now, we can discuss potential ways to fix this. The simple "solution" is to allow retries for this specific operation should the service return a 409. This also assumes, that the 409 is always being returned for this specific reason and not ever returned for any other reason. I don't know if this assumption can be made for this specific service operation.

Our Azure data-plane guidelines, were designed taking this race condition into account and so if the service were to implement this feature using Azure data-plane guidance, this problem is averted. Perhaps this operation can change its implementation to follow the data-plane guidance but the control-plane team would have to get involved in this and changing the behavior is a breaking change and Azure's policy very strongly discourages breaking changes.

The data-plane guidance disallows LRO PATCH operations for many reasons and you're experiencing one of them now. The data-plane guidance to enable this feature is to use an LRO POST-action pattern. I do think the control-plane should adopt this guidance too.

[horihiro]

Thank you Jeffrey Richter (@JeffreyRichter) for your explanation.

Because both of retrying for 409 response under a specific situation only and changing control-plane interface might not be practical, it's better that an application (GitHub Action in my case) retries to request in current status.

Is it appropriate to add a custom retry policy/strategy for 409 response to pipeline if the retry logic should be implemeted on application side?

[JeffreyRichter]

If you add retry on 409 to the pipeline (via client options) then this will apply to ALL operations on the client and this is not ideal. It may OK for this service but we definitely wouldn't do this for all services. What a customer could do is use one client for all operations except this one and use another client for this 1 operation where the retry policy is configured to retry on 409.

[JeffreyRichter]

Some language SDKs allow configuring the retry policy via client options. From the code you show above, it looks like JS does not support this. The JS SDK could add this feature and a customer could be instructed to add support for 409 to solve the problem in this issue. For example, our Go SDK lets the customer configure/override the HTTP status codes to retry on; see https://github.com/Azure/azure-sdk-for-go/blob/main/sdk/azcore/policy/policy.go#L106

The JS SDK should add this feature too (if not already there).