Inspektor Gadget is a collection of tools (or gadgets) to debug and inspect Kubernetes resources and applications. It manages the packaging, deployment and execution of eBPF programs in a Kubernetes cluster, including many based on BCC tools, as well as some developed specifically for use in Inspektor Gadget. It automatically maps low-level kernel primitives to high-level Kubernetes resources, making it easier and quicker to find the relevant information.
Inspektor Gadget tools are known as gadgets. You can deploy one, two or many gadgets.
Explore the following documentation to find out which tools can help you in your investigations.
Install Inspektor Gadget (client-side):
Use krew plugin manager to install:
$ kubectl krew install gadget
Install Inspektor Gadget on Kubernetes:
$ kubectl gadget deploy
Read the detailed install instructions to find more information.
kubectl gadget --help will provide you the list of supported commands and their flags.
$ kubectl gadget --help
Collection of gadgets for Kubernetes developers
advise Recommend system configurations based on collected information
audit Audit a subsystem
completion Generate the autocompletion script for the specified shell
deploy Deploy Inspektor Gadget on the cluster
help Help about any command
profile Profile different subsystems
prometheus Expose metrics using prometheus
run Run a containerized gadget (experimental)
script Run a bpftrace-compatible scripts
snapshot Take a snapshot of a subsystem and print it
sync Synchronize gadget information with server
top Gather, sort and periodically report events according to a given criteria
trace Trace and print system events
traceloop Get strace-like logs of a container from the past
undeploy Undeploy Inspektor Gadget from cluster
version Show version
You can then get help for each subcommand:
$ kubectl gadget advise --help
Recommend system configurations based on collected information
kubectl-gadget advise [command]
network-policy Generate network policies based on recorded network activity
seccomp-profile Generate seccomp profiles based on recorded syscalls activity
$ kubectl gadget audit --help
Audit a subsystem
kubectl-gadget audit [command]
seccomp Audit syscalls according to the seccomp profile
$ kubectl gadget profile --help
Profile different subsystems
kubectl-gadget profile [command]
block-io Analyze block I/O performance through a latency distribution
cpu Analyze CPU performance by sampling stack traces
tcprtt Analyze TCP connections through an Round-Trip Time (RTT) distribution
$ kubectl gadget snapshot --help
Take a snapshot of a subsystem and print it
kubectl-gadget snapshot [command]
process Gather information about running processes
socket Gather information about TCP and UDP sockets
$ kubectl gadget top --help
Gather, sort and periodically report events according to a given criteria
kubectl-gadget top [command]
block-io Periodically report block device I/O activity
ebpf Periodically report ebpf runtime stats
file Periodically report read/write activity by file
tcp Periodically report TCP activity
$ kubectl gadget trace --help
Trace and print system events
kubectl-gadget trace [command]
bind Trace socket bindings
capabilities Trace security capability checks
dns Trace DNS requests
exec Trace new processes
fsslower Trace open, read, write and fsync operations slower than a threshold
mount Trace mount and umount system calls
network Trace network streams
oomkill Trace when OOM killer is triggered and kills a process
open Trace open system calls
signal Trace signals received by processes
sni Trace Server Name Indication (SNI) from TLS requests
tcp Trace TCP connect, accept and close
tcpconnect Trace connect system calls
tcpdrop Trace TCP kernel-dropped packets/segments
tcpretrans Trace TCP retransmissions
Inspektor Gadget is deployed to each node as a privileged DaemonSet. It uses in-kernel eBPF helper programs to monitor events mainly related to syscalls from userspace programs in a pod. The eBPF programs are run by the kernel and gather the log data. Inspektor Gadget's userspace utilities fetch the log data from ring buffers and display it. What eBPF programs are and how Inspektor Gadget uses them is briefly explained in the architecture document.
Inspektor Gadget can also be used without Kubernetes to trace containers with
The different gadgets shipped with Inspektor Gadget use a variety of eBPF capabilities. The capabilities available depend on the version and configuration of the kernel running in the node. To be able to run all the gadgets, you'll need to have at least 5.10 with BTF enabled.
See requirements for a detailed list of the requirements per gadget.
There are some examples in this folder showing the usage
of the Golang packages provided by Inspektor Gadget. These examples are
designed for developers that want to use the Golang packages exposed by
Inspektor Gadget directly. End-users do not need this and can use
Contributions are welcome, see CONTRIBUTING.
We hold community meetings every other Thursday at 15:30 UTC, 7:30 PST, 16:30 CET in this link, check the calendar to have the full schedule of next meetings. Please add any topic you want to discuss to our meeting notes document.
Join the discussions on the
#inspektor-gadget channel in the Kubernetes Slack.
- Collecting Low-Level Metrics with eBPF, KubeCon + CloudNativeCon North America 2023 (video, slides)
- A (re)introduction of Inspektor Gadget: A Containerized Framework for eBPF Systems Inspection, Cloud Native Rejekts Chicago - November 2023 (video)
- Gaining Linux insights with Inspektor Gadget, an eBPF tool and systems inspection framework, All Systems Go - September 2023 (video)
- Overcoming the Challenges of Debugging Containers, Container Days Hamburg - September 2023 (video)
- Using the EBPF Superpowers To Generate Kubernetes Security Policies, KubeCon + CloudNativeCon North America 2022 (video, slides)
- Debug Your Clusters with eBPF-Powered Tools, Cloud Native eBPF Day North America 2022 (video, slides)
- Who Needs an API Server to Debug a Kubernetes Cluster?, Cloud Native eBPF Day North America 2022 (video, slides)
- Inspektor Gadget, introduction and demos, eCHO Livestream - September 2021 (video)
- OpenShift Commons Briefing: Unleash eBPF Superpowers with Kubectl Gadget, Openshift Commons 2020 (video)
- Tutorial: Understanding What Happens Inside Kubernetes Clusters Using BPF Tools, Open Source Summit EU 2020 (video)
- Inspektor Gadget and traceloop: Tracing containers syscalls using BPF, FOSDEM 2020 (video, slides)
- Traceloop for systemd and Kubernetes + Inspektor Gadget, All Systems Go 2019 (video)
- BPF Compiler Collection (BCC): some of the gadgets are based on BCC tools.
- kubectl-trace: the Inspektor Gadget architecture was inspired from kubectl-trace.
- cilium/ebpf: the gadget tracer manager and some other gadgets use the cilium/ebpf library.
The Inspektor Gadget user space components are licensed under the Apache License, Version 2.0. The BPF code templates are licensed under the General Public License, Version 2.0, with the Linux-syscall-note.