Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cilium configures multiple ipcache maps #25154

Open
joestringer opened this issue Apr 26, 2023 · 0 comments
Open

Cilium configures multiple ipcache maps #25154

joestringer opened this issue Apr 26, 2023 · 0 comments
Assignees
@ti-mo
Labels
sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. sig/loader Impacts the loading of BPF programs into the kernel.

Comments

@joestringer
Copy link
Member

During development on main during the v1.14 dev cycle, I noticed behaviour like this inside the cilium-agent container:

# bpftool map | grep -A 1 lpm
333: lpm_trie  name cilium_ipcache  flags 0x1
        key 24B  value 12B  max_entries 512000  memlock 20480000B
--
335: lpm_trie  name cilium_ipcache  flags 0x1
        key 24B  value 12B  max_entries 512000  memlock 20480000B
--
465: lpm_trie  name cilium_ipcache  flags 0x1
        key 24B  value 12B  max_entries 512000  memlock 20480000B
--
543: lpm_trie  name cilium_ipcache  flags 0x1
        key 24B  value 12B  max_entries 512000  memlock 20480000B

I've just changed the version to v1.13.2 and I'm observing similar behaviour. At a glance it seems like there are four of these maps during startup, then two of them get cleaned up and the other two remain around indefinitely.

There seems to be only one ipcache actually pinned to the filesystem:

# ls /sys/fs/bpf/tc/globals/*ipcache*
/sys/fs/bpf/tc/globals/cilium_ipcache
# bpftool map show pinned /sys/fs/bpf/tc/globals/cilium_ipcache
465: lpm_trie  name cilium_ipcache  flags 0x1
        key 24B  value 12B  max_entries 512000  memlock 20480000B

Particularly concerning is that it seems like some of the tail call programs are still referring to each ipcache map:

# bpftool prog | grep 465
        xlated 5792B  jited 3599B  memlock 8192B  map_ids 465,84,469,87,97
        xlated 1920B  jited 1235B  memlock 4096B  map_ids 465,84,97,87,469
        xlated 3912B  jited 2389B  memlock 4096B  map_ids 87,477,84,465,466,95,97
        xlated 3456B  jited 2057B  memlock 4096B  map_ids 87,95,465,477
        xlated 6456B  jited 3956B  memlock 8192B  map_ids 477,87,84,465,466,95,97
        xlated 3912B  jited 2389B  memlock 4096B  map_ids 87,478,84,465,466,95,97
        xlated 6456B  jited 3956B  memlock 8192B  map_ids 478,87,84,465,466,95,97
        xlated 9296B  jited 5256B  memlock 12288B  map_ids 465,87,559,95,94,124,98,99,96,561
        xlated 11552B  jited 6722B  memlock 12288B  map_ids 465,87,548,95,91,156,100,101,96,549
        xlated 11552B  jited 6722B  memlock 12288B  map_ids 465,87,560,95,91,120,100,101,96,565
        xlated 9296B  jited 5256B  memlock 12288B  map_ids 465,87,558,95,94,143,98,99,96,556
        xlated 13552B  jited 7842B  memlock 16384B  map_ids 465,567,131,95,98,99,94,96,87,568,84,97,466
        xlated 15856B  jited 9367B  memlock 16384B  map_ids 465,569,132,95,100,101,96,87,91,570,84,97,466
        xlated 9296B  jited 5256B  memlock 12288B  map_ids 465,87,563,95,94,120,98,99,96,565
        xlated 9296B  jited 5256B  memlock 12288B  map_ids 465,87,567,95,94,131,98,99,96,568
        xlated 13552B  jited 7842B  memlock 16384B  map_ids 465,559,124,95,98,99,94,96,87,561,84,97,466
        xlated 13552B  jited 7842B  memlock 16384B  map_ids 465,558,143,95,98,99,94,96,87,556,84,97,466
        xlated 11552B  jited 6722B  memlock 12288B  map_ids 465,87,564,95,91,124,100,101,96,561
        xlated 15856B  jited 9367B  memlock 16384B  map_ids 465,566,131,95,100,101,96,87,91,568,84,97,466
        xlated 15856B  jited 9367B  memlock 16384B  map_ids 465,553,143,95,100,101,96,87,91,556,84,97,466
        xlated 15856B  jited 9367B  memlock 16384B  map_ids 465,564,124,95,100,101,96,87,91,561,84,97,466
        xlated 15856B  jited 9367B  memlock 16384B  map_ids 465,560,120,95,100,101,96,87,91,565,84,97,466
        xlated 11552B  jited 6722B  memlock 12288B  map_ids 465,87,566,95,91,131,100,101,96,568
        xlated 15856B  jited 9367B  memlock 16384B  map_ids 465,548,156,95,100,101,96,87,91,549,84,97,466
        xlated 9296B  jited 5256B  memlock 12288B  map_ids 465,87,547,95,94,156,98,99,96,549
        xlated 13552B  jited 7842B  memlock 16384B  map_ids 465,563,120,95,98,99,94,96,87,565,84,97,466
        xlated 9296B  jited 5256B  memlock 12288B  map_ids 465,87,571,95,94,132,98,99,96,570
        xlated 13552B  jited 7842B  memlock 16384B  map_ids 465,547,156,95,98,99,94,96,87,549,84,97,466
        xlated 11552B  jited 6722B  memlock 12288B  map_ids 465,87,553,95,91,143,100,101,96,556
        xlated 13552B  jited 7842B  memlock 16384B  map_ids 465,571,132,95,98,99,94,96,87,570,84,97,466
        xlated 11552B  jited 6722B  memlock 12288B  map_ids 465,87,569,95,91,132,100,101,96,570
        xlated 13552B  jited 7842B  memlock 16384B  map_ids 465,574,474,95,98,99,94,96,87,573,84,97,466
3465: sched_cls  name tail_icmp6_send_time_exceeded  tag f5caf73db746c830  gpl
        xlated 15856B  jited 9367B  memlock 16384B  map_ids 465,572,474,95,100,101,96,87,91,573,84,97,466
        xlated 9296B  jited 5256B  memlock 12288B  map_ids 465,87,574,95,94,474,98,99,96,573
        xlated 11552B  jited 6722B  memlock 12288B  map_ids 465,87,572,95,91,474,100,101,96,573

# bpftool prog | grep 543
3450: sched_cls  name tail_icmp6_handle_ns  tag 277b36f3543e5cfb  gpl
        xlated 5792B  jited 3599B  memlock 8192B  map_ids 543,203,577,206,216
        xlated 1920B  jited 1235B  memlock 4096B  map_ids 543,203,216,206,577
        xlated 3912B  jited 2389B  memlock 4096B  map_ids 206,585,203,543,544,214,216
        xlated 6456B  jited 3956B  memlock 8192B  map_ids 585,206,203,543,544,214,216
        xlated 3456B  jited 2057B  memlock 4096B  map_ids 206,214,543,585
        xlated 6456B  jited 3956B  memlock 8192B  map_ids 586,206,203,543,544,214,216
        xlated 3912B  jited 2389B  memlock 4096B  map_ids 206,586,203,543,544,214,216
        xlated 9296B  jited 5256B  memlock 12288B  map_ids 543,206,591,214,213,231,217,218,215,590
        xlated 11552B  jited 6722B  memlock 12288B  map_ids 543,206,589,214,210,231,219,220,215,590
        xlated 15856B  jited 9367B  memlock 16384B  map_ids 543,589,231,214,219,220,215,206,210,590,203,216,544
3543: sched_cls  name tail_icmp6_handle_ns  tag 11ef2c320393207f  gpl
        xlated 13552B  jited 7842B  memlock 16384B  map_ids 543,591,231,214,217,218,213,215,206,590,203,216,544
        xlated 9296B  jited 5256B  memlock 12288B  map_ids 543,206,594,214,213,582,217,218,215,593
        xlated 11552B  jited 6722B  memlock 12288B  map_ids 543,206,592,214,210,582,219,220,215,593
        xlated 13552B  jited 7842B  memlock 16384B  map_ids 543,594,582,214,217,218,213,215,206,593,203,216,544
        xlated 15856B  jited 9367B  memlock 16384B  map_ids 543,592,582,214,219,220,215,206,210,593,203,216,544

This could lead to more severe connectivity issues if programs are relying on an ipcache map that is not getting updated.
@joestringer joestringer added sig/loader Impacts the loading of BPF programs into the kernel. sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. labels Apr 26, 2023
@ti-mo ti-mo self-assigned this Apr 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ti-mo ti-mo

Labels
sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. sig/loader Impacts the loading of BPF programs into the kernel.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@joestringer @ti-mo