REST: Add retry and timeout configuration for REST catalog#3418
Open
rahulsmahadev wants to merge 1 commit into
Open
REST: Add retry and timeout configuration for REST catalog#3418rahulsmahadev wants to merge 1 commit into
rahulsmahadev wants to merge 1 commit into
Conversation
Closes apache#2772. The REST Catalog uses requests with no retries and no timeout by default, so transient 5xx/network failures bubble up immediately and slow servers can hang the client indefinitely (e.g. a Polaris instance returning 504 from a proxy). Add an optional connection: block on the catalog properties: catalog: default: uri: http://rest-catalog/ws/ connection: timeout: 60 retry: total: 5 backoff_factor: 1.0 status_forcelist: [429, 500, 502, 503, 504] allowed_methods: [GET, HEAD, OPTIONS] connection.retry is passed verbatim to urllib3.util.retry.Retry. Both keys are optional and opt-in: when neither is set the default requests behavior is preserved. Signed-off-by: rahulsmahadev <rahul.mahadev@databricks.com>
Contributor
|
Can you take a look at the linter failures? 'make lint' will help you run the linters locally. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
The REST Catalog uses
requestswith no retries and no timeout configured, so transient 5xx / network failures bubble up immediately and slow servers can hang the client indefinitely. The reporting user on #2772 hit infinite-timeout hangs against a Polaris instance returning504from a proxy.This change adds an optional
connection:block on the catalog properties for opting in to a per-request timeout and a retry policy, as proposed in the issue body:Changes:
connection,connection.timeout,connection.retrycatalog properties._RetryTimeoutHTTPAdapterthat subclassesrequests.adapters.HTTPAdapterand applies a default timeout to every call (sincerequestsdoes not expose a session-wide timeout).connection.retrysub-mapping is passed verbatim tourllib3.util.retry.Retry, so any kwarg the upstream class accepts can be configured.requestsbehavior is preserved — no behavior change for existing users.timeoutraisesValueError; unknownRetrykwargs are caught and re-raised asValueErrorwith a clearer message.The SigV4 adapter mounted at the catalog URI (see #3063) is a longer-prefix match and still wins for that host, so SigV4 users continue to get their existing retry behavior unchanged — this change deliberately stays out of that path.
Are these changes tested?
Yes. Added five tests in
tests/catalog/test_rest.py:test_session_without_connection_config_uses_default_adapter— noconnection:block → no_RetryTimeoutHTTPAdapteris mounted (defaultrequestsbehavior preserved).test_session_with_connection_timeout_and_retry— full nestedconnection:block applies timeout and allRetrykwargs (total,backoff_factor,status_forcelist,allowed_methods).test_session_with_connection_timeout_only—timeoutalone works without aretryblock.test_session_with_invalid_connection_timeout_raises— negativetimeoutraisesValueError.test_session_with_invalid_connection_retry_kwarg_raises— unknownRetrykwarg raisesValueError.Are there any user-facing changes?
Yes — two new optional REST catalog properties (
connection.timeout,connection.retry) documented inmkdocs/docs/configuration.mdunder "Retry and timeout". No behavior change for users who do not set them.