loader: explicitly insert endpoint policy programs into policy maps

[ti-mo]

BPF ELF loaders generally populate maps in a non-deterministic order when it comes to the order of the maps themselves. During some loads, cilium_calls may be populated first, sometimes cilium_(egress)call_policy gets populated first. The latter case is a problem since existing endpoints will constantly get their cilium_call_policy entry invoked for incoming packets. If its tail call map(s) are not yet populated, that will result in missed tail calls leading to packet drops.

This commit zeroes the call maps' MapSpec.Contents and manually extracts the endpoint id from the ELF section name of the policy progs, both for ingress and egress (l7) handlers.

This calls for major refactoring -- inserting the policy program is equal to attaching an entrypoint to a BPF hook and should be done explicitly by an endpoint manager that has access to the Endpoint object. This would make extracting the endpoint ID from the ELF section name redundant.

Fixes: #27720
Fixes: #26739

Avoid missed tail calls due to inserting policy programs too early during endpoint regeneration

@joe @jrajahalme MBOI

[ti-mo]

/test

[ti-mo]

/test

[ti-mo]

/test

[brb]

During some loads, cilium_calls may be populated first, sometimes cilium_(egress)call_policy gets populated first. The latter case is a problem since existing endpoints will constantly get their cilium_call_policy entry invoked for incoming packets. If its tail call map(s) are not yet populated, that will result in missed tail calls leading to packet drops.

@ti-mo Could you elaborate more? In particular, is the following correct?

1. During reloading, cilium_call_policy global map is recreated. Same goes for cilium_calls global map.
2. The new cilium_calls references the new cilium_call_policy.
3. The new cilium_call_policy entries get populated during the async endpoint regeneration.
4. The new cilium_calls map is used by reloaded bpf_host which might do a tail call via the new cilium_call_policy[ into yet not populated entry.

Your change changes the 3. step in a way that the cilium_call_policy get populated before referenced by the cilium_calls. Is it correct?

both for ingress and egress (l7) handlers

What do you mean by "(l7)"?

[ti-mo]

@brb Thanks for the questions!

1. During reloading, cilium_call_policy global map is recreated. Same goes for cilium_calls global map.

cilium_(egress)call_policy are not recreated unless they're resized (based on ENDPOINTS_MAP_SIZE). They're shared across bpf_host, bpf_overlay and bpf_lxc, so resizing means we need to replace all Cilium progs on the system. Currently there's no mechanism that tries to make this seamless (we don't claim it to be), since the agent doesn't explicitly manage these maps. After the first ELF with updated ENDPOINTS_MAP_SIZE is loaded, the old one will be unpinned, removing all endpoints' entries. It's undefined behaviour what will and won't work, this is out of scope of this PR/issue.

cilium_calls_01234 is per-endpoint/device and not global.

2. The new cilium_calls references the new cilium_call_policy.

There's only supposed to be one cilium_(egress)call_policy, so both old and new endpoint progs will point to the same during endpoint regenerations.

3. The new cilium_call_policy entries get populated during the async endpoint regeneration.

Not completely sure what you mean by this, but this patch makes it so insertions into cilium_(egress)call_policy are delayed until after the ELF's entrypoints are attached.

4. The new cilium_calls map is used by reloaded bpf_host which might do a tail call via the new cilium_call_policy[ into yet not populated entry.

Not sure if still relevant due to points above.

Your change changes the 3. step in a way that the cilium_call_policy get populated before referenced by the cilium_calls. Is it correct?

The other way around! cilium_calls is internal to the ELF and needs to be fully populated in order for the programs' control flow to be sound. When this is completed, entrypoints can be attached and cilium_(egress)call_policy can be upserted.

both for ingress and egress (l7) handlers

What do you mean by "(l7)"?

I'm not quite sure what cilium_egresscall_policy's purpose is, but my understanding is these programs are called for packets egressing Envoy. Someone else might need to fill in the gaps here.

[brb]

Thanks for the explanation and the patch! LGTM.

[brb]

This calls for major refactoring -- inserting the policy program is equal to attaching an entrypoint to a BPF hook and should be done explicitly by an endpoint manager that has access to the Endpoint object. This would make extracting the endpoint ID from the ELF section name redundant.

@ti-mo 👍 Let's create an issue for it, so that we could prioritize it for v1.16.