Envoy Load Shedding && Maintenance Mode

[justConfused]

Title: How to load shed in envoy to help upstream service recover

Description:

Trying to figure out how to configure load shading to help upstream service recover. Currently we are using envoy for s2s communication. I am not sure is load shedding is meant to achieved via circuit breaking or maintenance mode.
Also I am having hard time find documentation on maintenance mode how to enable, disable and configure it. Plus how does envoy behave in this mode.

Really the appreciate the help.

PS: Envoy is a great tool and its community is equally awesome.

[jmarantz]

Hi @justConfused -- there are 2 related subsystems at least. One primarily protects upstreams:: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/circuit_breaking

The other primarily protects Envoy:
https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/overload_manager

What is maintenance mode? I'm not familiar with that Envoy concept yet.

[justConfused]

Hi @jmarantz

I would like start up by mentioning, I am very new to envoy. I have been following a talk by @ccaraman

Reducing Mean-Time-to-Detection of Incidents with an Envoy Service Mesh
https://www.youtube.com/watch?v=q_psmHeXaGI

In the talk she mentions putting envoy in maintenance_mode which allowed them to shed load to a service by percentage. It looks to be disjoint from circuit_breakers.

I would like to learn more about this mode and its usage. Following is the relevant documentation I have found. Looking for something more in-depth possibly with an example.

https://www.envoyproxy.io/docs/envoy/latest/configuration/http_filters/router_filter#config-http-filters-router-runtime-maintenance-mode

[mattklein123]

The above documentation points to how to use runtime (typically via a file system) to have Envoy automatically respond with a 503 (known as "maintenance mode"). There is also a plan to have this be part of the config that can be served via EDS, where it is known as "drop overload" but the concept is the same: #1355. Today this is only doable via runtime. (Though technically it could be accomplished via a direct response route injected by RDS).

[justConfused]

Thanks @mattklein123 this better helps me under this options.