faultInjectionOnGateway #35378

xinlian12 · 2023-06-08T16:07:57Z

Description

Added capability for fault injection in gateway layer + staled_addresses in rntbd layer.
FaultInjection in rntbd layer - #33329

Design

High level (FaultInjection on gateway)

From public API perspective, there are no major changes. Customer can config fault injection behavior by creating FaultInjectionRule. Each Rule contains major three components (condition, result, id/lifecycle). Please see more details here #33329. However, there are few more constraints about the error scenarios being supported.

result: Type of FaultInjectionServerErrorResult

No FaultInjectionConnectionErrorResult will be allowed to be configured on gateway. Java SDK internally use netty http client which will manage the lifecycles of all connections, so we will just treat it as a black box.

FaultInjectionResultBuilders
    .getResultBuilder(FaultInjectionServerErrorType.TOO_MANY_REQUEST)
    .times(1)
    .build()

Supported server error type:

RESPONSE_DELAY, // can be used to simulate transient timeout/broken connections
CONNECTION_DELAY, // can be used to simulate network issue/connection timeout
TOO_MANY_REQUEST,
READ_SESSION_NOT_AVAILABLE,
TIMEOUT,
RETRY_WITH,
INTERNAL_SERVER_ERROR,
SERVICE_UNAVAILABLE

Note: GONE is not supported on gateway as gateway will retry, so SDK will get 503 instead

Supported operation type:

    READ_ITEM,
    QUERY_ITEM,
    CREATE_ITEM,
    UPSERT_ITEM,
    REPLACE_ITEM,
    DELETE_ITEM,
    PATCH_ITEM,
   * METADATA_REQUEST_CONTAINER,
   * METADATA_REQUEST_DATABASE_ACCOUNT,
   * METADATA_REQUEST_PARTITION_KEY_RANGES,
   * METADATA_REQUEST_REFRESH_ADDRESSES,
   * METADATA_REQUEST_QUERY_PLAN

OperationType	GONE	RESPONSE_DELAY	CONNECTION_DELAY	TOO_MANY_REQUEST	READ_SESSION_NOT_AVAILABLE	TIMEOUT	RETRY_WITH	INTERNAL_SERVER_ERROR	SERVICE_UNAVAILABLE
*_ITEM	✖️	✔️	✔️	✔️	✔️	✔️	✔️	✔️	✔️
METADATA_REQUEST_*	✖️	✔️	✔️	✔️	✖️	✖️	✖️	✖️	✖️

Name	Notes
`times`	Optional. Limit how many times the same rule can be applied on the operation. For metadata request rule, if the metadata request is initiated by/attached to another data plane operation, it essentially means how many times the data plane operation will observe the metadata related errors.
`delay`	Required for `RESPONSE_DELAY`, `CONNECTION_DELAY`
`suppressServiceRequests`	Only used for `RESPONSE_DELAY`, it controls when injected delay > network request timeout, whether the request will still be sent to sever. It will be useful for testing retriable writes testing

condition: Type of FaultInjectionCondition. A rule will only be applied if all the necessary conditions matches.

new FaultInjectionConditionBuilder()
     .connectionType(FaultInjectionConnectionType.GATEWAY)
     .region("East US 2")
     .operationType(FaultInjectionOperationType.METADATA_REQUEST_ADDRESS_REFRESH)
     .endpoints(
          new FaultInjectionEndpointBuilder(FeedRange.forLogicalPartition(new PartitionKey("Test"))).build()
     )
     .build();

Name	Default	Notes
`OperationType`	`Null`	`Optional`. Will be ignored for `SERVER_CONNECTION_DELAY`
`ConnectionType`	`FaultInjectionConnectionType.DIRECT`	For rule configured with metadata operationType, the connection type will be defaulted to `FaultInjectionConnectionType.GATEWAY`. For rule on gateway please use `FaultInjectionConnectionType.GATEWAY`
`Region`	`Null`	`Optional`. If not defined, the rule will be applied in all available regions
`Endpoints`	`Null`	`Optional`. Type of `FaultInjectionEndpoint`. Use when you want to filter down to certain partition. Ignored for metadata operation type except `METADATA_REQUEST_REFRESH_ADDRESSES`. Ignored for error type `CONNECTION_DELAY`

id/duration/hitLimit/startDelay: The unique identifier of the rule and the effective life span of the rule.

Diagnostics

faultInjectionRuleId and faultInjectionEvaluationResults will be included in the CosmosDiagnostics for easy debugging

Testing scenario example

Network issue/Connection timeout for address refresh requests

FaultInjectionRule faultInjectionRule = new FaultInjectionRuleBuilder("addressRefreshDelay")
            .condition(
                new FaultInjectionConditionBuilder()
                    .region("EAST US 2")
                    .connectionType(FaultInjectionConnectionType.GATEWAY)
                    .operationType(FaultInjectionOperationType.METADATA_REQUEST_REFRESH_ADDRESSES)
                    .build())
            .result(
                FaultInjectionResultBuilders.getResultBuilder(FaultInjectionServerErrorType.CONNECTION_DELAY)
                    .delay(Duration.ofSeconds(60))
                    .build()
            )
            .build();

Broken connections

FaultInjectionRule faultInjectionRule = new FaultInjectionRuleBuilder("addressRefreshDelay")
            .condition(
                new FaultInjectionConditionBuilder()
                    .region("EAST US 2")
                    .connectionType(FaultInjectionConnectionType.GATEWAY)
                    .operationType(FaultInjectionOperationType.METADATA_REQUEST_REFRESH_ADDRESSES)
                    .build())
            .result(
                FaultInjectionResultBuilders.getResultBuilder(FaultInjectionServerErrorType.RESPONSE_DELAY)
                    .delay(Duration.ofSeconds(5))
                    .build()
            )
            .build();

Server return Too_Many_Request exception for all partitions

FaultInjectionRule faultInjectionRule = new FaultInjectionRuleBuilder("addressRefreshDelay")
            .condition(
                new FaultInjectionConditionBuilder()
                    .region("EAST US 2")
                    .connectionType(FaultInjectionConnectionType.GATEWAY)
                    .operationType(FaultInjectionOperationType.METADATA_REQUEST_REFRESH_ADDRESSES)
                    .build())
            .result(
                FaultInjectionResultBuilders.getResultBuilder(FaultInjectionServerErrorType.TOO_MANY_REQUEST)
                    .times(3)
                    .build()
            )
            .build();

Server return Too_Many_Request exception for one partition

FaultInjectionRule faultInjectionRule = new FaultInjectionRuleBuilder("addressRefreshDelay")
            .condition(
                new FaultInjectionConditionBuilder()
                    .region("EAST US 2")
                    .connectionType(FaultInjectionConnectionType.GATEWAY)
                    .operationType(FaultInjectionOperationType.METADATA_REQUEST_REFRESH_ADDRESSES)
                    .endpoints(
                        new FaultInjectionEndpointBuilder(FeedRange.forLogicalPartition(new PartitionKey("Test"))).build()
                    )
                    .build())
            .result(
                FaultInjectionResultBuilders.getResultBuilder(FaultInjectionServerErrorType.TOO_MANY_REQUEST)
                    .times(3)
                    .build()
            )
            .build();

High level (STALED_ADDRESSES_SERVER_GONE)

This error type is used to simulate request failed with GONE error due to SDK has used staled addresses (which could be due to many reasons, partition split/merge/migrated etc). Unlike other exception where you can control how many times the error can be injected, STALED_ADDRESSES_SERVER_GONE will only be cleared if a forceRefresh address refresh happened.

During the effectiveness lifespan of the rule, for each operation will go through the following cycle

  graph LR
  A[Request] --> B{Match fault injection rule}
  B -- Yes --> C[Inject GONE exception]
  B -- No -->  D[Original flow]
  C --> E[Retry]
  E --> F{Has forceRefresh address happened}
  F -- Yes --> D
  F --No --> C

Other changes included

currently gatewayDiagnostics only track the diagnostics for the last retry/try. In this PR, it changed to use gatewayDiagnosticsList to track all details.

azure-sdk · 2023-06-08T16:16:51Z

API change check

APIView has identified API level changes in this PR and created following API reviews.

azure-cosmos

xinlian12 · 2023-07-07T06:55:27Z

/azp run java - cosmos - tests

azure-pipelines · 2023-07-07T06:55:39Z

Azure Pipelines successfully started running 1 pipeline(s).

xinlian12 · 2023-07-07T14:52:40Z

/azp run java - cosmos - tests

azure-pipelines · 2023-07-07T14:52:54Z

Azure Pipelines successfully started running 1 pipeline(s).

FabianMeiswinkel

LGTM - Thanks

xinlian12 · 2023-07-11T20:20:35Z

/azp run java - cosmos - tests

azure-pipelines · 2023-07-11T20:20:41Z

No commit pushedDate could be found for PR 35378 in repo Azure/azure-sdk-for-java

xinlian12 · 2023-07-11T20:31:16Z

/azp run java - cosmos - tests

azure-pipelines · 2023-07-11T20:31:29Z

Azure Pipelines successfully started running 1 pipeline(s).

kushagraThapar

LGTM, thanks @xinlian12

sdk/cosmos/azure-cosmos-test/CHANGELOG.md

...osmos-test/src/main/java/com/azure/cosmos/test/faultinjection/FaultInjectionRuleBuilder.java

...in/java/com/azure/cosmos/test/implementation/faultinjection/FaultInjectionRuleProcessor.java

...t/src/main/java/com/azure/cosmos/test/implementation/faultinjection/ServerErrorInjector.java

sdk/cosmos/azure-cosmos/CHANGELOG.md

kushagraThapar · 2023-07-11T22:10:16Z

sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/RxDocumentClientImpl.java

        }
+
+        this.gatewayProxy.configureFaultInjectorProvider(injectorProvider, this.configs);


This will always be configured now, even if the fault injector is in direct mode, right?

yes, because no matter whether gateway or direct, there are always requests going to gateway

sdk/cosmos/azure-cosmos/src/main/java/com/azure/cosmos/implementation/RxGatewayStoreModel.java

...os/src/main/java/com/azure/cosmos/implementation/directconnectivity/GatewayAddressCache.java

...c/main/java/com/azure/cosmos/implementation/faultinjection/FaultInjectionRequestContext.java

xinlian12 · 2023-07-12T04:14:50Z

/azp run java - cosmos - tests

azure-pipelines · 2023-07-12T04:14:55Z

No commit pushedDate could be found for PR 35378 in repo Azure/azure-sdk-for-java

xinlian12 · 2023-07-12T05:52:36Z

/azp run java - cosmos - tests

azure-pipelines · 2023-07-12T05:52:49Z

Azure Pipelines successfully started running 1 pipeline(s).

faultInjectionOnGateway

0dcc767

github-actions bot added the Cosmos label Jun 8, 2023

annie-mac added 12 commits June 8, 2023 13:10

fix failed test

1c14e30

change

fc24a64

Merge branch 'main' into faultInjectionOnGateway

5693fbb

Merge branch 'main' into faultInjectionOnGateway

c28f7be

Merge branch 'main' into faultInjectionOnGateway

cebd12d

refactor

efa00c4

Merge branch 'main' into faultInjectionOnGateway

729001b

refactor

80d3c15

Merge branch 'main' into faultInjectionOnGateway

9bf249f

wire up metada fault injection context

8221619

revert back

b19a31b

add tests

68d00b1

fix

389431e

annie-mac added 2 commits July 7, 2023 09:03

update change log

38e5ac1

fix tests

3fec2de

xinlian12 marked this pull request as ready for review July 7, 2023 17:23

xinlian12 requested review from kushagraThapar, FabianMeiswinkel, kirankumarkolli, milismsft, aayush3011, simorenoh and jeet1995 as code owners July 7, 2023 17:23

xinlian12 requested a review from Pilchie as a code owner July 7, 2023 17:23

change

2a7b2d9

FabianMeiswinkel approved these changes Jul 10, 2023

View reviewed changes

annie-mac added 2 commits July 11, 2023 11:52

Merge branch 'main' into faultInjectionOnGateway

69cc7e4

fix javadoc

52a140d

kushagraThapar approved these changes Jul 11, 2023

View reviewed changes

xinlian12 force-pushed the faultInjectionOnGateway branch from 87305e3 to 4c85a01 Compare July 12, 2023 03:57

resolve comments

7d11985

xinlian12 force-pushed the faultInjectionOnGateway branch from 4c85a01 to 7d11985 Compare July 12, 2023 04:13

xinlian12 merged commit 40e0e3f into Azure:main Jul 12, 2023
50 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

faultInjectionOnGateway #35378

faultInjectionOnGateway #35378

xinlian12 commented Jun 8, 2023 •

edited by FabianMeiswinkel

Loading

azure-sdk commented Jun 8, 2023 •

edited

Loading

xinlian12 commented Jul 7, 2023

azure-pipelines bot commented Jul 7, 2023

xinlian12 commented Jul 7, 2023

azure-pipelines bot commented Jul 7, 2023

FabianMeiswinkel left a comment

xinlian12 commented Jul 11, 2023

azure-pipelines bot commented Jul 11, 2023

xinlian12 commented Jul 11, 2023

azure-pipelines bot commented Jul 11, 2023

kushagraThapar left a comment

kushagraThapar Jul 11, 2023

xinlian12 Jul 11, 2023

xinlian12 commented Jul 12, 2023

azure-pipelines bot commented Jul 12, 2023

xinlian12 commented Jul 12, 2023

azure-pipelines bot commented Jul 12, 2023

		}

		this.gatewayProxy.configureFaultInjectorProvider(injectorProvider, this.configs);

faultInjectionOnGateway #35378

faultInjectionOnGateway #35378

Conversation

xinlian12 commented Jun 8, 2023 • edited by FabianMeiswinkel Loading

Description

Design

High level (FaultInjection on gateway)

result: Type of FaultInjectionServerErrorResult

Supported server error type:

Supported operation type:

condition: Type of FaultInjectionCondition. A rule will only be applied if all the necessary conditions matches.

id/duration/hitLimit/startDelay: The unique identifier of the rule and the effective life span of the rule.

Diagnostics

Testing scenario example

Network issue/Connection timeout for address refresh requests

Broken connections

Server return Too_Many_Request exception for all partitions

Server return Too_Many_Request exception for one partition

High level (STALED_ADDRESSES_SERVER_GONE)

Other changes included

azure-sdk commented Jun 8, 2023 • edited Loading

xinlian12 commented Jul 7, 2023

azure-pipelines bot commented Jul 7, 2023

xinlian12 commented Jul 7, 2023

azure-pipelines bot commented Jul 7, 2023

FabianMeiswinkel left a comment

Choose a reason for hiding this comment

xinlian12 commented Jul 11, 2023

azure-pipelines bot commented Jul 11, 2023

xinlian12 commented Jul 11, 2023

azure-pipelines bot commented Jul 11, 2023

kushagraThapar left a comment

Choose a reason for hiding this comment

kushagraThapar Jul 11, 2023

Choose a reason for hiding this comment

xinlian12 Jul 11, 2023

Choose a reason for hiding this comment

xinlian12 commented Jul 12, 2023

azure-pipelines bot commented Jul 12, 2023

xinlian12 commented Jul 12, 2023

azure-pipelines bot commented Jul 12, 2023

xinlian12 commented Jun 8, 2023 •

edited by FabianMeiswinkel

Loading

`result`: Type of `FaultInjectionServerErrorResult`

`condition`: Type of `FaultInjectionCondition`. A rule will only be applied if all the necessary conditions matches.

`id`/`duration`/`hitLimit`/`startDelay`: The unique identifier of the rule and the effective life span of the rule.

azure-sdk commented Jun 8, 2023 •

edited

Loading