API and implementation to quarantine service backends #18814

aditighag · 2022-02-15T17:46:11Z

The API will allow users to set backend states. The primary motivation is to quarantine unreachable backends so that such
backends won't be selected for load-balancing service traffic.
The backend states can be set in response to various events like Kubernetes events (e.g., terminating endpoints), or service APIs (e.g., quarantine unreachable backends). Backend states are persisted across cilium agent restart.

$ cilium service list
ID   Frontend          Service Type   Backend
4    10.96.95.246:80   ClusterIP      1 => 10.244.0.58:80 (active)
                                      2 => 10.244.1.59:80 (active)
                                      3 => 10.244.1.8:80 (active)

$ cilium service update --backends 10.244.0.58:80 --states quarantined
Updating backend states
Updated service with 1 backends

$ cilium service list
ID   Frontend          Service Type   Backend
4    10.96.95.246:80   ClusterIP      1 => 10.244.0.58:80 (quarantined)
                                      2 => 10.244.1.59:80 (active)
                                      3 => 10.244.1.8:80 (active)

$ cilium service update --backends 10.244.0.58:80 --states active
Updating backend states
Updated service with 1 backends

$ cilium service list
ID   Frontend          Service Type   Backend
4    10.96.95.246:80   ClusterIP      1 => 10.244.0.58:80 (active)
                                      2 => 10.244.1.59:80 (active)
                                      3 => 10.244.1.8:80 (active)

See commits description for more details.

Support setting service backend states such as quarantine, maintenance so that these backends are not selected for load-balancing service traffic.

aditighag · 2022-02-15T22:47:10Z

/test

aditighag · 2022-02-15T23:14:42Z

/ci-l4lb

aditighag · 2022-02-23T06:36:46Z

/test

aditighag · 2022-03-01T00:54:33Z

/ci-l4lb

pkg/k8s/watchers/watcher.go

aditighag · 2022-03-04T02:18:53Z

/test

The API will allow users to update backend states. The primary motivation is to quarantine unreachable backends so that such backends won't be selected for load-balancing service traffic. Internally, the agent iterate over the services map, and update all the services that select the updated backends. States transitions are internally validated before updating the states. See dump below to how to set "quarantine" state for backends. Quarantined backends when become reachable can be set back to the active state. ``` $ cilium service list ID Frontend Service Type Backend 4 10.96.95.246:80 ClusterIP 1 => 10.244.0.58:80 (active) 2 => 10.244.1.59:80 (active) 3 => 10.244.1.8:80 (active) $ cilium service update --backends 10.244.0.58:80 --states quarantined Updating backend states Updated service with 1 backends $ cilium service list ID Frontend Service Type Backend 4 10.96.95.246:80 ClusterIP 1 => 10.244.0.58:80 (quarantined) 2 => 10.244.1.59:80 (active) 3 => 10.244.1.8:80 (active) $ cilium service update --backends 10.244.0.58:80 --states active Updating backend states Updated service with 1 backends $ cilium service list ID Frontend Service Type Backend 4 10.96.95.246:80 ClusterIP 1 => 10.244.0.58:80 (active) 2 => 10.244.1.59:80 (active) 3 => 10.244.1.8:80 (active) $ cilium service update --backends 10.244.0.58:80 --states terminating Updating backend states Updated service with 1 backends e$ cilium service list ID Frontend Service Type Backend 4 10.96.95.246:80 ClusterIP 1 => 10.244.0.58:80 (terminating) 2 => 10.244.1.59:80 (active) 3 => 10.244.1.8:80 (active) $ cilium service update --backends 10.244.0.58:80 --states active Updating backend states Error: Cannot add/update service: [PUT /service/{id}][501] putServiceIdUpdateBackendFailure invalid state transition for backend[10.244.0.58:80] (terminating) -> (active) &{BackendAddresses:[0xc0000f8990] Flags:0xc0001004b0 FrontendAddress:0xc00012f9c0 ID:0} ``` Signed-off-by: Aditi Ghag <aditi@cilium.io>

Use the padding bits as flags. This is a preparatory commit for the subsequent commits to be able to encode backend state in the flags field which can be restored after agent restart. Signed-off-by: Aditi Ghag <aditi@cilium.io>

The `flags` field in the backend map value will be used to encode the backend's state. This helps in restoring backend states after agent restart. Signed-off-by: Aditi Ghag <aditi@cilium.io>

Backend states can change until the backends are deleted. The new states need to be updated in the BPF backend map so that the last updated state can be restored after agent restart. ``` $ cilium service list ID Frontend Service Type Backend 4 10.96.95.246:80 ClusterIP 1 => 10.244.0.58:80 (active) 2 => 10.244.1.59:80 (active) 3 => 10.244.1.8:80 (active) $ cilium service update --backends 10.244.0.58:80 --states quarantined Updating backend states Updated service with 1 backends $ sudo bpftool map dump pinned /sys/fs/bpf/tc/globals/cilium_lb4_backends_v2 key: 03 00 00 00 value: 0a f4 01 08 00 50 00 00 key: 02 00 00 00 value: 0a f4 01 3b 00 50 00 00 key: 01 00 00 00 value: 0a f4 00 3a 00 50 00 02 $ cilium service update --backends 10.244.0.58:80 --states active Updating backend states Updated service with 1 backends $ cilium service list ID Frontend Service Type Backend 4 10.96.95.246:80 ClusterIP 1 => 10.244.0.58:80 (active) 2 => 10.244.1.59:80 (active) 3 => 10.244.1.8:80 (active) $ sudo bpftool map dump pinned /sys/fs/bpf/tc/globals/cilium_lb4_backends_v2 key: 03 00 00 00 value: 0a f4 01 08 00 50 00 00 key: 02 00 00 00 value: 0a f4 01 3b 00 50 00 00 key: 01 00 00 00 value: 0a f4 00 3a 00 50 00 00 Found 3 elements ``` Signed-off-by: Aditi Ghag <aditi@cilium.io>

This commit builds up on previous commits where we restore non-active backends after agent restart. The non-active backend states such as `Quarantined` will not be synced with Kubernetes events as these are set with the Cilium service API. As a result, we need to persist backend state in BPF maps, and restore it after agent restart. When the agent is restarted, we first restore backends and services from BPF maps. The persisted backends are marked as restored from datapath. Later, once the agent's sync with Kubernetes API server is complete, we reconcile the state restored from BPF maps with the service and endpoint related events received from Kubernetes. The backend flag that tracks restored backends from datapath is used to restore backend state during this phase. Testing - Before agent restart - ``` $ cilium service list ID Frontend Service Type Backend 4 10.96.95.246:80 ClusterIP 1 => 10.244.0.58:80 (active) 2 => 10.244.1.59:80 (active) 3 => 10.244.1.8:80 (active) $ sudo bpftool map dump pinned /sys/fs/bpf/tc/globals/cilium_lb4_backends_v2 key: 03 00 00 00 value: 0a f4 01 08 00 50 00 00 key: 02 00 00 00 value: 0a f4 01 3b 00 50 00 00 key: 01 00 00 00 value: 0a f4 00 3a 00 50 00 00 $ cilium service update --backends 10.244.0.58:80 --states quarantined Updating backend states Updated service with 1 backends $ sudo bpftool map dump pinned /sys/fs/bpf/tc/globals/cilium_lb4_backends_v2 key: 03 00 00 00 value: 0a f4 01 08 00 50 00 00 key: 02 00 00 00 value: 0a f4 01 3b 00 50 00 00 key: 01 00 00 00 value: 0a f4 00 3a 00 50 00 02 Found 3 elements $ sudo systemctl restart cilium ``` After restart - ``` $ sudo bpftool map dump pinned /sys/fs/bpf/tc/globals/cilium_lb4_backends_v2 key: 03 00 00 00 value: 0a f4 01 08 00 50 00 00 key: 02 00 00 00 value: 0a f4 01 3b 00 50 00 00 key: 01 00 00 00 value: 0a f4 00 3a 00 50 00 02 Found 3 elements $ cilium service list ID Frontend Service Type Backend 4 10.96.95.246:80 ClusterIP 1 => 10.244.0.58:80 (quarantined) 2 => 10.244.1.59:80 (active) 3 => 10.244.1.8:80 (active) ``` Signed-off-by: Aditi Ghag <aditi@cilium.io>

Validates service API to update backend states reflects updated state in services list. Also, tests restore workflow after agent restart. Signed-off-by: Aditi Ghag <aditi@cilium.io>

aditighag · 2022-04-21T06:25:16Z

All tests except the vagrant ones have passed. Latest commits include a simple change - #18814 (comment).

aditighag · 2022-04-21T06:25:56Z

/test

Job 'Cilium-PR-K8s-GKE' failed:

Click to show.

Test Name

K8sHealthTest cilium-health Checks status between nodes

Failure Output

FAIL: Expected

If it is a flake and a GitHub issue doesn't already exist to track it, comment /mlh new-flake Cilium-PR-K8s-GKE so I can create one.

aditighag · 2022-04-21T18:34:12Z

/test-1.23-net-next

aditighag · 2022-04-21T18:34:22Z

/test-1.21-5.4

aditighag · 2022-04-21T21:03:32Z

/test-vagrant

aditighag · 2022-04-21T21:04:33Z

/test-gke

Previous - https://jenkins.cilium.io/job/Cilium-PR-K8s-GKE/8244/testReport/junit/Suite-k8s-1/20/K8sHealthTest_cilium_health_Checks_status_between_nodes/
Same as #18258.

aditighag · 2022-04-21T21:25:14Z

/test-gke

The previous test phrase didn't re-trigger the job for some reason.

aditighag · 2022-04-21T22:06:51Z

/test-1.21-5.4

Previous -

14:42:56      k8s1-1.21: E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 5238 (apt-get)
14:42:56      k8s1-1.21: E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?

Job 'Cilium-PR-K8s-GKE' failed:

Click to show.

Test Name

K8sHealthTest cilium-health Checks status between nodes

Failure Output

FAIL: Expected

If it is a flake and a GitHub issue doesn't already exist to track it, comment /mlh new-flake Cilium-PR-K8s-GKE so I can create one.

aditighag · 2022-04-21T23:02:03Z

/test-gke

Job 'Cilium-PR-K8s-1.23-kernel-net-next' failed:

Click to show.

Test Name

K8sUpdates Tests upgrade and downgrade from a Cilium stable image to master

Failure Output

FAIL: terminating containers are not deleted after timeout

If it is a flake and a GitHub issue doesn't already exist to track it, comment /mlh new-flake Cilium-PR-K8s-1.23-kernel-net-next so I can create one.

aditighag · 2022-04-22T03:38:44Z

/test-runtime

aditighag · 2022-04-22T03:39:38Z

net-next hit #18873.

Use the backend 'flags' field instead the old 'pad'. This fixes bpf compilation fails when L7 LB is used, like this: ``` level=warning msg="/var/lib/cilium/bpf/bpf_sock.c:387:14: error: no member named 'pad' in 'struct lb4_backend'" subsys=datapath-loader level=warning msg=" l7backend.pad = 0;" subsys=datapath-loader level=warning msg=" ~~~~~~~~~ ^" subsys=datapath-loader level=warning msg="1 error generated." subsys=datapath-loader ``` Add ENABLE_L7_LB to LB_OPTIONS in bpf/Makefile to catch this kind of errors in future. Fixes: cilium#18814 Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>

Use the backend 'flags' field instead the old 'pad'. This fixes bpf compilation fails when L7 LB is used, like this: ``` level=warning msg="/var/lib/cilium/bpf/bpf_sock.c:387:14: error: no member named 'pad' in 'struct lb4_backend'" subsys=datapath-loader level=warning msg=" l7backend.pad = 0;" subsys=datapath-loader level=warning msg=" ~~~~~~~~~ ^" subsys=datapath-loader level=warning msg="1 error generated." subsys=datapath-loader ``` Add ENABLE_L7_LB to LB_OPTIONS in bpf/Makefile to catch this kind of errors in future. Fixes: #18814 Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>

aditighag added the release-note/minor This PR changes functionality that users may find relevant to operating Cilium. label Feb 15, 2022

aditighag requested review from a team February 15, 2022 17:46

aditighag requested a review from a team as a code owner February 15, 2022 17:46

aditighag requested review from borkmann, sayboras and nathanjsweet February 15, 2022 17:46

maintainer-s-little-helper bot assigned borkmann, sayboras and nathanjsweet Feb 15, 2022

aditighag marked this pull request as draft February 15, 2022 17:46

aditighag force-pushed the pr/aditighag/quarantine-lb-backends-api branch from 05ea342 to 89816f4 Compare February 15, 2022 23:13

aditighag force-pushed the pr/aditighag/quarantine-lb-backends-api branch from 89816f4 to ce8955d Compare February 23, 2022 06:36

aditighag marked this pull request as ready for review February 25, 2022 19:51

aditighag requested a review from a team as a code owner February 25, 2022 19:51

aditighag marked this pull request as draft February 28, 2022 22:57

aditighag force-pushed the pr/aditighag/quarantine-lb-backends-api branch from ce8955d to 56c1d7d Compare March 1, 2022 00:53

borkmann reviewed Mar 1, 2022

View reviewed changes

pkg/k8s/watchers/watcher.go Show resolved Hide resolved

pchaigno unassigned nathanjsweet, borkmann and sayboras Mar 2, 2022

aditighag mentioned this pull request Mar 2, 2022

maglev: support setting a weight of a backend in a service spec via new cmdline argument #18306

Merged

aditighag force-pushed the pr/aditighag/quarantine-lb-backends-api branch from 56c1d7d to 6d2fdc9 Compare March 4, 2022 02:13

aditighag marked this pull request as ready for review March 4, 2022 02:19

aditighag force-pushed the pr/aditighag/quarantine-lb-backends-api branch from 6d2fdc9 to 6399f12 Compare March 4, 2022 02:29

aditighag added 6 commits April 20, 2022 14:27

bpf,lb: Define flags in backend map value

0b8d388

Use the padding bits as flags. This is a preparatory commit for the subsequent commits to be able to encode backend state in the flags field which can be restored after agent restart. Signed-off-by: Aditi Ghag <aditi@cilium.io>

lb: Encode backend state in backend map value

44aaea6

The `flags` field in the backend map value will be used to encode the backend's state. This helps in restoring backend states after agent restart. Signed-off-by: Aditi Ghag <aditi@cilium.io>

test: Add runtime test to validate backend state updates

49a6808

Validates service API to update backend states reflects updated state in services list. Also, tests restore workflow after agent restart. Signed-off-by: Aditi Ghag <aditi@cilium.io>

aditighag added release-note/major This PR introduces major new functionality to Cilium. and removed release-note/minor This PR changes functionality that users may find relevant to operating Cilium. labels Apr 21, 2022

aditighag force-pushed the pr/aditighag/quarantine-lb-backends-api branch from be39ab6 to 49a6808 Compare April 21, 2022 06:25

sayboras mentioned this pull request Apr 21, 2022

clustermesh: Add support for service-affinity #19521

Merged

19 tasks

aditighag added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Apr 22, 2022

tklauser merged commit 4b52130 into cilium:master Apr 22, 2022

jrajahalme mentioned this pull request Apr 22, 2022

bpf/sock: Use renamed field #19532

Merged

ysksuzuki mentioned this pull request Jul 6, 2022

bpf: Handle tuple collisions for inactive backends #20407

Merged

nbusseneau mentioned this pull request Aug 9, 2022

v1.11 backports 2022-08-09 #20840

Merged

zuzzas mentioned this pull request Aug 24, 2022

Do not enable health checks on Terminating backends #21062

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API and implementation to quarantine service backends #18814

API and implementation to quarantine service backends #18814

aditighag commented Feb 15, 2022 •

edited

Loading

aditighag commented Feb 15, 2022

aditighag commented Feb 15, 2022

aditighag commented Feb 23, 2022

aditighag commented Mar 1, 2022

aditighag commented Mar 4, 2022

aditighag commented Apr 21, 2022

aditighag commented Apr 21, 2022 •

edited by maintainer-s-little-helper bot

Loading

Test Name

Failure Output

aditighag commented Apr 21, 2022

aditighag commented Apr 21, 2022

aditighag commented Apr 21, 2022

aditighag commented Apr 21, 2022 •

edited

Loading

aditighag commented Apr 21, 2022 •

edited

Loading

aditighag commented Apr 21, 2022 •

edited by maintainer-s-little-helper bot

Loading

Test Name

Failure Output

aditighag commented Apr 21, 2022 •

edited by maintainer-s-little-helper bot

Loading

Test Name

Failure Output

aditighag commented Apr 22, 2022

aditighag commented Apr 22, 2022

API and implementation to quarantine service backends #18814

API and implementation to quarantine service backends #18814

Conversation

aditighag commented Feb 15, 2022 • edited Loading

aditighag commented Feb 15, 2022

aditighag commented Feb 15, 2022

aditighag commented Feb 23, 2022

aditighag commented Mar 1, 2022

aditighag commented Mar 4, 2022

aditighag commented Apr 21, 2022

aditighag commented Apr 21, 2022 • edited by maintainer-s-little-helper bot Loading

Test Name

Failure Output

aditighag commented Apr 21, 2022

aditighag commented Apr 21, 2022

aditighag commented Apr 21, 2022

aditighag commented Apr 21, 2022 • edited Loading

aditighag commented Apr 21, 2022 • edited Loading

aditighag commented Apr 21, 2022 • edited by maintainer-s-little-helper bot Loading

Test Name

Failure Output

aditighag commented Apr 21, 2022 • edited by maintainer-s-little-helper bot Loading

Test Name

Failure Output

aditighag commented Apr 22, 2022

aditighag commented Apr 22, 2022

aditighag commented Feb 15, 2022 •

edited

Loading

aditighag commented Apr 21, 2022 •

edited by maintainer-s-little-helper bot

Loading

aditighag commented Apr 21, 2022 •

edited

Loading

aditighag commented Apr 21, 2022 •

edited

Loading

aditighag commented Apr 21, 2022 •

edited by maintainer-s-little-helper bot

Loading

aditighag commented Apr 21, 2022 •

edited by maintainer-s-little-helper bot

Loading