Adding abort-percentage and abort-http-status HTTP filters

[donyu]

Title: Add capability to control abort-percentage & abort-http-status settings using HTTP headers

Context: At Lyft, we have plans to run client-side fault injection tests. These tests would allow engineers to gain visibility into how our apps actually behave in degenerate conditions such as backend service mesh outages and connectivity loss. Some example use cases include:

• Simulating client-side app behavior when a series of endpoints or service goes down.
• Identifying client error-handling issues for specific request error status code.

Proposal:

Our plan is to add the following capabilities to http/fault/fault_filter.cc:

• Specify a % of ingress requests that should be aborted
• Specify the HTTP status code that should be returned for aborted requests

These capabilities are already available as fault injection runtime settings. We have an example of similar settings being controlled using HTTP headers - see x-envoy-fault-throughput-response and x-envoy-fault-delay-request HTTP headers that were implemented as part of this Github Issue.

We propose to add support for the following two HTTP headers:

HTTP Header	Type	Associated Runtime Setting
x-envoy-fault-abort-percentage	int (percentage between 0 and 100)	fault.http.abort.abort_percent
x-envoy-fault-abort-http-status	int (status codes e.g. 504, 404, etc.)	fault.http.abort.http_status

[mattklein123]

Thanks for opening! Whatever we implement we should definitely do as part of the current filter filter.

My only real question is whether the percentage header as value? Couldn't the client just calculate its own percentage? Or do you want this for simplicity? I don't feel strongly about it one way or the other.

[donyu]

Thanks for the feedback! I updated the issue to reflect only adding to the current fault filter.

Yes, the client can calculate the fault-abort-percentage on its own. We wanted to expose this header for both simplicity (less client-side logic) and to ensure platform consistency.

[mattklein123]

Yes, the client can calculate the fault-abort-percentage on its own. We wanted to expose this header for both simplicity (less client-side logic) and to ensure platform consistency.

SGTM

[Augustyniak]

we can default to 100% in case x-envoy-fault-abort-percentage HTTP header is missing and x-envoy-fault-abort-http-status is set and valid (http status code).

[Augustyniak]

The support for x-envoy-fault-abort-request HTTP header was added in #10294.

It's x-envoy-fault-abort-request and not x-envoy-fault-abort-http-status as it was originally proposed since x-envoy-fault-abort-request seems to follow the existing HTTP header scheme names better (prior to the referenced change Envoy supported x-envoy-fault-delay-request and x-envoy-fault-throughput-response headers).

[Augustyniak]

I think that we can close this issue now. The support for x-envoy-fault-abort-request header was added in #10294 (per my message above) and I opened a separate issue for the proposal of addition of 3 HTTP headers that would allow us to control the percentage of requests faults are injected into - the issue can be found here #10648.