-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
faultInjectionOnGateway #35378
faultInjectionOnGateway #35378
Conversation
API change check APIView has identified API level changes in this PR and created following API reviews. |
/azp run java - cosmos - tests |
Azure Pipelines successfully started running 1 pipeline(s). |
/azp run java - cosmos - tests |
Azure Pipelines successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM - Thanks
/azp run java - cosmos - tests |
No commit pushedDate could be found for PR 35378 in repo Azure/azure-sdk-for-java |
/azp run java - cosmos - tests |
Azure Pipelines successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks @xinlian12
...osmos-test/src/main/java/com/azure/cosmos/test/faultinjection/FaultInjectionRuleBuilder.java
Show resolved
Hide resolved
...in/java/com/azure/cosmos/test/implementation/faultinjection/FaultInjectionRuleProcessor.java
Outdated
Show resolved
Hide resolved
...t/src/main/java/com/azure/cosmos/test/implementation/faultinjection/ServerErrorInjector.java
Outdated
Show resolved
Hide resolved
} | ||
|
||
this.gatewayProxy.configureFaultInjectorProvider(injectorProvider, this.configs); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will always be configured now, even if the fault injector is in direct mode, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, because no matter whether gateway or direct, there are always requests going to gateway
sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/RxGatewayStoreModel.java
Outdated
Show resolved
Hide resolved
...os/src/main/java/com/azure/cosmos/implementation/directconnectivity/GatewayAddressCache.java
Outdated
Show resolved
Hide resolved
...os/src/main/java/com/azure/cosmos/implementation/directconnectivity/GatewayAddressCache.java
Show resolved
Hide resolved
...c/main/java/com/azure/cosmos/implementation/faultinjection/FaultInjectionRequestContext.java
Outdated
Show resolved
Hide resolved
87305e3
to
4c85a01
Compare
4c85a01
to
7d11985
Compare
/azp run java - cosmos - tests |
No commit pushedDate could be found for PR 35378 in repo Azure/azure-sdk-for-java |
/azp run java - cosmos - tests |
Azure Pipelines successfully started running 1 pipeline(s). |
Description
Added capability for fault injection in gateway layer + staled_addresses in rntbd layer.
FaultInjection in rntbd layer - #33329
Design
High level (FaultInjection on gateway)
From public API perspective, there are no major changes. Customer can config fault injection behavior by creating
FaultInjectionRule
. Each Rule contains major three components (condition, result, id/lifecycle). Please see more details here #33329. However, there are few more constraints about the error scenarios being supported.No
FaultInjectionConnectionErrorResult
will be allowed to be configured on gateway. Java SDK internally use netty http client which will manage the lifecycles of all connections, so we will just treat it as a black box.Supported server error type:
Note: GONE is not supported on gateway as gateway will retry, so SDK will get 503 instead
Supported operation type:
times
delay
RESPONSE_DELAY
,CONNECTION_DELAY
suppressServiceRequests
RESPONSE_DELAY
, it controls when injected delay > network request timeout, whether the request will still be sent to sever. It will be useful for testing retriable writes testingOperationType
Null
Optional
. Will be ignored forSERVER_CONNECTION_DELAY
ConnectionType
FaultInjectionConnectionType.DIRECT
FaultInjectionConnectionType.GATEWAY
. For rule on gateway please useFaultInjectionConnectionType.GATEWAY
Region
Null
Optional
. If not defined, the rule will be applied in all available regionsEndpoints
Null
Optional
. Type ofFaultInjectionEndpoint
. Use when you want to filter down to certain partition. Ignored for metadata operation type exceptMETADATA_REQUEST_REFRESH_ADDRESSES
. Ignored for error typeCONNECTION_DELAY
faultInjectionRuleId
andfaultInjectionEvaluationResults
will be included in theCosmosDiagnostics
for easy debuggingNetwork issue/Connection timeout for address refresh requests
Broken connections
Server return Too_Many_Request exception for all partitions
Server return Too_Many_Request exception for one partition
High level (STALED_ADDRESSES_SERVER_GONE)
This error type is used to simulate request failed with GONE error due to SDK has used staled addresses (which could be due to many reasons, partition split/merge/migrated etc). Unlike other exception where you can control how many times the error can be injected, STALED_ADDRESSES_SERVER_GONE will only be cleared if a forceRefresh address refresh happened.
During the effectiveness lifespan of the rule, for each operation will go through the following cycle
Other changes included
gatewayDiagnostics
only track the diagnostics for the last retry/try. In this PR, it changed to usegatewayDiagnosticsList
to track all details.