enhancement: oauth2 filter supports retries to improve communication stability with OAuth2 severs

[zhaohuabing]

Title: enhancement: oauth2 filter supports retries to improve communication stability with OAuth2 severs

Description:
EG user reported: we are frequently seeing routes hosted by the Gateway fail with a 401, returning an error OAuth flow failed.. Looking at the logs it seems this occurs where Envoy decides to re-use an existing long-lived connection to our OIDC issuer. Failures seem to correlate with Envoy picking connections that are in the order of 50 minutes old.

Allowing retries after failing to send requests to OAuth2 server can solve this problem.

[optional Relevant Links:]
EG issue: envoyproxy/gateway#3178

[zhaohuabing]

Please assign this issue to me if it makes sense.

[adisuissa]

cc @derekargueta @mattklein123 as codeowners

[derekargueta]

So the issue is that the connection may have been closed by the identity provider but the Oauth filter still tries the old connection? If that's the case a retry mechanism sounds pretty reasonable to me.

Wonder if there's a deeper issue here of the closed connection being recognized by the oauth filter (unless it's a race condition as mentioned in the EG ticket)

[zhaohuabing]

@derekargueta I found some similar issues. They didn't happen in the Oauth filter but I think they're still valid. This issue is caused by the HTTP Client using a closed connection from the pool.
#14981
istio/istio#47037

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[derekargueta]

@zhaohuabing are you still planning on providing a PR for this issue?

[zhaohuabing]

@derekargueta yes, please assign it to me. Thanks!