BPF & XDP for containers
Cilium provides fast in-kernel networking and security policy enforcement for containers based on eBPF programs generated on the fly. It is an experimental project aiming at enabling emerging kernel technologies such as BPF and XDP for containers.
- Cilium Daemon: Agent written in Go. Generates & compiles the BPF programs, manages the BPF maps, and interacts with the local container runtime.
- BPF programs:
- container: Container connectivity
- netdev: Integration with L3 networks (physical/virtual)
- overlay: Integration with overlay networks (VXLAN, Geneve)
- load balancer: Fast L3/L4 load balancer with direct server return.
- Integration: CNI, Kubernetes, Docker
- 5-min Quickstart: Using the prebuilt docker images
- For Developers: Setting up a vagrant environment
- Manual installation: Detailed installation instructions
- F.A.Q.: F.A.Q.
The following are video tutorials showcasing how to use Cilium:
- Networks & simple policies
- Debugging a connectivity issue
- Examine networking configuration of container
What is eBPF and XDP?
Berkley Packet Filter (BPF) is a bytecode interpreter orignially introduced to filter network packets, e.g. tcpdump and socket filters. It has since been extended to with additional data structures such as hashtable and arrays as well as additional actions to support packet mangling, forwarding, encapsulation, etc. An in-kernel verifier ensures that BPF programs are safe to run and a JIT compiler converts the bytecode to CPU architecture specifc instructions for native execution efficiency. BPF programs can be run at various hooking points in the kernel such as for incoming packets, outgoing packets, system call level, kprobes, etc.
XDP is a further step in evolution and enables to run a specific flavour of BPF programs from the network driver with direct access to the packet's DMA buffer.
What are the benefits of Cilium's use of BPF?
- simple: Every container is assigned a unique IPv6 address. An IPv4 address can be assigned optionally. There is no concept of networks, all containers are connected to a single virtual space. Isolation among containers is defined based on container labels.
- ipv6-focused IPv6 is considered the primary addressing model with IPv4 support provided for backwards compatibility based on either native integration or with NAT46.
- extendable: Users can extend and customize any aspect of the BPF programs. Forwarding logic and policy enforcement is not limited to the capabilities of a specific Linux kernel version. This may include the addition of additional statistics not provided by the Linux kernel, support for additional protocol parsers, modifications of the connection tracker or policy layer, additional forwarding logic, etc.
- fast: The BPF JIT compiler integrated into the Linux kernel guarantees for efficient execution of BPF programs. A separate BPF program is generated for each individual container on the fly which allows to automatically reduce the code size to the minimal, similar to static linking.
- hotfixable: Updates to the kernel forwarding path can be applied without restarting the kernel or any of the running containers.
- debuggable: A highly efficient monitoring subsystem is integrated and can be enabled on demand at runtime. It provides visibility into the network activity of containers under high network speeds without disruption or introduction of latency.
Cilium requires a recent version of the Linux kernel, iproute2 and clang+LLVM. All required changes have been merged upstream and are available in official releases:
- Linux >= 4.8.0: http://www.kernel.org/
- iproute2 >= 4.8.0: https://www.kernel.org/pub/linux/utils/net/iproute2/
- clang+LLVM >=3.7.1: http://llvm.org/releases
Cilium can make use of additional functionality available in >= 4.9 kernels. It will probe for the availability upon startup and enable it automatically. Development snapshots of the Linux kernel and iproute2 tree can be found here:
Alternatively, the vagrant box
noironetworks/net-next is built regularly
and provides the above branches compiled and pre-installed. See the
vagrant section for instructions on how to bring up a
virtual machine with all dependencies pre installed.
See the installation instructions.
Cilium provides integration plugins for the following orchestration systems:
We are eager to receive feedback and contributions. Please see the contributing guide for further instructions and ideas on how to contribute.
- Docker Distributed Systems Summit, Berlin, Oct 2016: Slides, Video
- NetDev1.2, Tokyo, Sep 2016 - cls_bpf/eBPF updates since netdev 1.1: Slides, Video
- NetDev1.2, Tokyo, Sep 2016 - Advanced programmability and recent updates with tc’s cls_bpf: Slides, Video
- ContainerCon NA, Toronto, Aug 2016 - Fast IPv6 container networking with BPF & XDP: Slides
- NetDev1.1, Seville, Feb 2016 - On getting tc classifier fully programmable with cls_bpf: Slides, Video
- Software Gone Wild by Ivan Pepelnjak, Oct 2016: Blog, MP3
- OVS Orbit by Ben Pfaff, May 2016: Blog, MP3
- Cilium, BPF and XDP, Google Open Source Blog, Nov 2016: Blog
If you have any questions feel free to contact us on Slack