Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Proxy: Dedicated DaemonSet for Envoy Proxy #25076

Closed
wants to merge 17 commits into from
Closed

Proxy: Dedicated DaemonSet for Envoy Proxy #25076

wants to merge 17 commits into from

Conversation

mhofstetter
Copy link
Member

@mhofstetter mhofstetter commented Apr 24, 2023 •
edited

Currently, whenever L7 capabilities are required, the Cilium L7 Proxy (Envoy) gets executed as separate process within the CIlium Agent k8s container.

This PR adds support for deploying Envoy independently as a separate K8s DaemonSet for availability, performance, and (potentially ) security benefits.

Potential Benefits

  • Restarting cilium-agent for upgrades does not impact live dataplane traffic (this is a property of cilium in general for the BPF datapath, but no longer applies once any envoy-based forwarding is brought in).
  • Failures of cilium-agent to start that result in cilium pods in crashloop backoff do not impact dataplane traffic.
  • Performance isolation between envoy and cilium-agent. An overloaded envoy will not swamp the cilium-agent, or vice versa. Can also have separate CPU/mem limits.

Tasks

  • Docker Image Build for Cilium Proxy
  • Make targets for building and publishing Cilium Proxy to local k8s clusters
  • Build & Publish Cilium Proxy images in CI (quay.io/cilium/proxy)
  • Separate directory for Envoy proxy sockets (admin, xDS, accesslogger) -> will be shared via host between agent & proxy
  • Agent Configuration, whether proxy should be started in embedded mode or not (will default to embedded for the time being)
  • Introduce CIlium Proxy DaemonSet in Cilium Helm Chart
  • Bugtool: adapt changed admin api socket path when gathering Envoy Config during Cilium SysDump
  • Add Metrics Listener as static resource when deploying proxy as DaemonSet
  • Use /ready endpoint as health endpoint for Cilium Proxy Pods
  • Envoy Version check via Admin Interface instead of checking via binary version output
  • Configurable Envoy application log configuration (log path, log format, log level) -> Envoy Application Log will only be visible in CIlium Proxy container
  • Temporary GitHub action based on Conformance Kind for testing Cilium connectivity tests against cilium with proxy in DaemonSet

This PR tries to cover the relevant tasks to provide support for deploying Envoy in its own DaemonSet - without getting too big. The following tasks were out of scope.

  • Optimize runtime base image of Proxy (currently still builds on top of the same base image as the agent)
  • Reduction of the permission of the Agent DaemonSet deployment
  • Resource configuration for the Proxy DaemonSet deployment
  • Optimized behaviour if Proxy isn't deployed
  • ...

Please review the individual commits! They are more or less aligned with above tasks.

Installation

For the time being, the default will still be the "embedded" envoy proxy. Therefore, Proxy as DaemonSet must be explicitly enabled via helm value proxy.enabled=true.

  • Helm: supported
  • Cilium CLI: out of the box supported via newly introduced "helm mode" (requires latest Cilium CLI from main branch).

ℹ️ Adaptions (status, sysdump, install) in cilium/cilium-cli will follow soon

Release Note

Support for deploying Cilium L7 Proxy (Envoy) independently as a separate daemonset for availability, performance, and security benefits.

@mhofstetter mhofstetter added area/proxy Impacts proxy components, including DNS, Kafka, Envoy and/or XDS servers. release-note/major This PR introduces major new functionality to Cilium. area/servicemesh GH issues or PRs regarding servicemesh labels Apr 24, 2023
@mhofstetter mhofstetter force-pushed the pr/mhofstetter/envoy-daemonset branch from 130fc1c to fe4844b Compare April 24, 2023 08:48
@mhofstetter
images: introduce Dockerfile for Cilium Proxy image
b056a1a 
This commit introduces the Dockerfile for building the Cilium Proxy
image. For now, it builds on top of the runtime image and copies the
proxy binary from github.com/cilium/proxy Docker Image.

Signed-off-by: Marco Hofstetter <marco.hofstetter@isovalent.com>
@mhofstetter
Makefile: add targets to build Cilium Proxy image
fd5bc70 
This commit adds the necessary Make targets to build the Cilium Proxy
Docker image.

Signed-off-by: Marco Hofstetter <marco.hofstetter@isovalent.com>
@mhofstetter
ci: build Cilium Proxy docker image in github
b7e2938 
This commit adds building the Cilium Proxy Docker Image to the GitHub
Action.

Signed-off-by: Marco Hofstetter <marco.hofstetter@isovalent.com>
@mhofstetter
proxy: separate directory for proxy sockets
7d7bef4 
Currently, the /var/run/cilium is used as state and runtime directory
for various aspects. The Unix Sockets for the communciation between
Cilium Agent & Cilium Proxy (Envoy) (xDS, Admin, AccessLog)are created
in this directory too.

With the need of sharing the sockets between Agent & Proxy Pod via
hostmount in the future, this commit moves the sockets into the
directory /var/run/proxy.

Signed-off-by: Marco Hofstetter <marco.hofstetter@isovalent.com>
@mhofstetter
proxy: introduce configuration for starting Proxy in embedded mode
600807c 
Currently, the Cilium Proxy is started as standalone process embedded
within the Cilium Agent Pod.

During the transition towards having a dedicated DaemonSet for the
Cilium Proxy, this commit introduces the possibility to configure
whether the embedded proxy should be enabled or not via Cilium config
property `proxy-embedded-enabled`.

Signed-off-by: Marco Hofstetter <marco.hofstetter@isovalent.com>
@mhofstetter
proxy: envoy admin client refactoring
ebc1995 
This commit refactors the initialization of the embedded Envoy Proxy
instance by introducing a EnvoyAdminClient interface which encapsulates
the communication with the Envoy Proxy instance - whether this is the
embedded or standalone (DaemonSet) instance.

Signed-off-by: Marco Hofstetter <marco.hofstetter@isovalent.com>
@mhofstetter mhofstetter force-pushed the pr/mhofstetter/envoy-daemonset branch from fe4844b to ee534fd Compare April 24, 2023 08:49
@mhofstetter
install: add Cilium Proxy DaemonSet to Cilium Helm Chart
45477ae 
This commit introduces the DaemonSet for the Cilium Proxy.

Signed-off-by: Marco Hofstetter <marco.hofstetter@isovalent.com>
@mhofstetter
kind: adapt local image to kind helm values
3132b96 
With this commit, the Helm values for local kind environments is adapted
to support the locally built proxy image.

Signed-off-by: Marco Hofstetter <marco.hofstetter@isovalent.com>
@mhofstetter
bugtool: change socket path for envoy admin socket
daee9bf 
This commit adapts the change of the envoy unix socket location when
dumping the Envoy config & metrics. Unix Sockets are now located within
`/var/run/proxy`.

Signed-off-by: Marco Hofstetter <marco.hofstetter@isovalent.com>
@mhofstetter
envoy: metrics listener as static resource for Proxy DaemonSet
08d1435 
Currently, the metrics listener is added dynamically at runtime - mainly
to prevent the embedded Envoy from not starting if the port is already
used.

In case of an dedicated Proxy DaemonSet, the metrics listener is gets
configured as static resource within the bootstrap config.

Signed-off-by: Marco Hofstetter <marco.hofstetter@isovalent.com>
@mhofstetter
proxy: use /ready from Envoy admin interface for health probes
80f97fa 
With this commit, the Proxy DaemonSet uses the "/ready" from the Envoy
admin interface for health probes (startup, live & ready).

Therefore, it resuses the listener of the metrics listener - mainly to
prevent the need for another port.

Signed-off-by: Marco Hofstetter <marco.hofstetter@isovalent.com>
@mhofstetter
proxy: Envoy version check via admin interface during initialization
e99a38f 
Currently, the Envoy version of the Cilium proxy gets checked at startup
via the binaries version command.

Due to extraction of the Cilium Proxy into its own DaemonSet, this
commits retrieves the version via Envoys admin interface. Therefore,
version check happens after initializing the EnvoyAdminClient.

Signed-off-by: Marco Hofstetter <marco.hofstetter@isovalent.com>
@mhofstetter
proxy: Adapt envoy log path for Proxy DaemonSet
c75e17a 
Currently, the embedded Envoy Proxy instance supports changing the log
path from stdout to a file via cilium config property `envoy-log`.

This commit introduces the Helm property `proxy.log.path` which
configures the Cilium property and the Cilium DaemonSet start arguments.

Signed-off-by: Marco Hofstetter <marco.hofstetter@isovalent.com>
@mhofstetter
proxy: Configurable Envoy log format via Helm Values
5eaef12 
Currently, the embedded Envoy Proxy instance hardcodes Envoys
application log format to an adapted form of the Cilium log format and
switches to the default Envoy log format if a log path is defined.

By having the Cilium Proxy in a dedicated DaemonSet, the default is the
Envoy log format.

In addition, this commit provides the possibiliity to configure the log
format via Helm value `proxy.log.format`.

Signed-off-by: Marco Hofstetter <marco.hofstetter@isovalent.com>
@mhofstetter
proxy: Adapt envoy log level to Proxy DaemonSet
6b092ba 
This commit adapts Envoys Log Level in the Proxy DaemonSet to use the
debug & debug-verbose options from the Helm Chart.

* "debug.verbose" contains "envoy" -> trace
* "debug.enabled" & "debug.verbose" contains "flow" -> debug
* default -> info

Signed-off-by: Marco Hofstetter <marco.hofstetter@isovalent.com>
@mhofstetter
proxy: move embedded envoy aspects into dedicated file
e2e54e9 
This commit refactors all aspects of the embedded Envoy proxy instance
into a separate file `embedded_envoy.go` and separates it from other
aspects like EnvoyAdminClient.

Signed-off-by: Marco Hofstetter <marco.hofstetter@isovalent.com>
@mhofstetter mhofstetter force-pushed the pr/mhofstetter/envoy-daemonset branch 2 times, most recently from 488a4c9 to 853fae3 Compare April 24, 2023 08:56
@mhofstetter
ci: temporary GHA Conformance Kind test with proxy daemonset
33ffea8 
Signed-off-by: Marco Hofstetter <marco.hofstetter@isovalent.com>
@mhofstetter mhofstetter force-pushed the pr/mhofstetter/envoy-daemonset branch from 853fae3 to 33ffea8 Compare April 24, 2023 08:57
@sayboras sayboras self-requested a review April 24, 2023 10:07
@mhofstetter
Copy link
Member Author

closed in favor of #25081

@mhofstetter mhofstetter removed the request for review from sayboras April 24, 2023 10:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
area/proxy Impacts proxy components, including DNS, Kafka, Envoy and/or XDS servers. area/servicemesh GH issues or PRs regarding servicemesh release-note/major This PR introduces major new functionality to Cilium.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@mhofstetter