Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: Print previous logs for gadget pods #2389

Merged
merged 1 commit into from
Jan 22, 2024
Merged

CI: Print previous logs for gadget pods #2389

merged 1 commit into from
Jan 22, 2024

Conversation

mauriciovasquezbernal
Copy link
Member

We're missing the logs if a gadget pod crashes. Use --previous to print logs if the pod had crashed.

It happened for instance in https://github.com/inspektor-gadget/inspektor-gadget/actions/runs/7572632543/attempts/1#summary-20624243194.

command.go:273: Inspektor Gadget pod logs:
command.go:274: + kubectl get pods -n gadget -o wide
NAME           READY   STATUS    RESTARTS      AGE    IP           NODE                      NOMINATED NODE   READINESS GATES
gadget-d4z7x   1/1     Running   1 (39s ago)   109s   10.244.1.2   minikube-containerd-m02   <none>           <none>
gadget-sx6nw   1/1     Running   0             109s   10.244.0.3   minikube-containerd       <none>           <none>
+ kubectl get pods -n gadget -o name
+ kubectl logs -n gadget pod/gadget-d4z7x
time="2024-01-18T16:34:03Z" level=info msg="OS detected: Ubuntu 20.04.5 LTS"
time="2024-01-18T16:34:03Z" level=info msg="Kernel detected: 6.2.0-1018-azure"
time="2024-01-18T16:34:03Z" level=info msg="Gadget Image: ghcr.io/inspektor-gadget/inspektor-gadget:latest"
time="2024-01-18T16:34:03Z" level=info msg="Deployment options:"
time="2024-01-18T16:34:03Z" level=info msg="INSPEKTOR_GADGET_OPTION_HOOK_MODE=auto"
time="2024-01-18T16:34:03Z" level=info msg="INSPEKTOR_GADGET_OPTION_FALLBACK_POD_INFORMER=true"
time="2024-01-18T16:34:03Z" level=info msg="Inspektor Gadget version: d7f610b"
time="2024-01-18T16:34:03Z" level=info msg="Gadget Tracer Manager hook mode: auto"
time="2024-01-18T16:34:03Z" level=info msg="Starting the Gadget Tracer Manager..."
time="2024-01-18T16:34:03Z" level=info msg="Experimental features enabled"
time="2024-01-18T16:34:03Z" level=info msg="HostPID=false"
time="2024-01-18T16:34:03Z" level=info msg="HostNetwork=false"
time="2024-01-18T16:34:04Z" level=info msg="GadgetTracerManager: hook mode: fanotify+ebpf (auto)"
time="2024-01-18T16:34:04Z" level=info msg="GadgetTracerManager: enabling fallback podinformer"
time="2024-01-18T16:34:04Z" level=info msg="Starting Pod controller"
time="2024-01-18T16:34:04Z" level=info msg="gadget tracermgr set in kubemanager"
time="2024-01-18T16:34:04Z" level=info msg="Serving on gRPC socket /run/gadgettracermanager.socket"
time="2024-01-18T16:34:04Z" level=warning msg="container gadget/gadget-d4z7x/gadget wasn't detected by the main hook! The fallback pod informer will add it."
time="2024-01-18T16:34:04Z" level=error msg="cgroup enricher: failed to get cgroup paths on container f86ad0205fdbf06ee232322c5806087d871ce6b224436df8583cc71e8be98a83: cgroup path not found in /proc/PID/cgroup"
time="2024-01-18T16:34:05Z" level=info msg="Starting trace controller manager"
time="2024-01-18T16:34:05Z" level=info msg="Reconcile trace gadget/traceloop-k8x4j (gadget traceloop, node minikube-containerd-m02)"
time="2024-01-18T16:34:05Z" level=info msg="No annotations. Nothing to do."
time="2024-01-18T16:34:10Z" level=info msg="Trace \"gadget/network-graph-2xsjg\" has been deleted"
+ kubectl logs -n gadget pod/gadget-sx6nw
time="2024-01-18T16:32:54Z" level=info msg="OS detected: Ubuntu 20.04.5 LTS"
time="2024-01-18T16:32:54Z" level=info msg="Kernel detected: 6.2.0-1018-azure"
time="2024-01-18T16:32:54Z" level=info msg="Gadget Image: ghcr.io/inspektor-gadget/inspektor-gadget:latest"
time="2024-01-18T16:32:54Z" level=info msg="Deployment options:"
time="2024-01-18T16:32:54Z" level=info msg="INSPEKTOR_GADGET_OPTION_HOOK_MODE=auto"
time="2024-01-18T16:32:54Z" level=info msg="INSPEKTOR_GADGET_OPTION_FALLBACK_POD_INFORMER=true"
time="2024-01-18T16:32:54Z" level=info msg="Inspektor Gadget version: d7f610b"
time="2024-01-18T16:32:54Z" level=info msg="Gadget Tracer Manager hook mode: auto"
time="2024-01-18T16:32:54Z" level=info msg="Starting the Gadget Tracer Manager..."
time="2024-01-18T16:32:54Z" level=info msg="Experimental features enabled"
time="2024-01-18T16:32:54Z" level=info msg="HostPID=false"
time="2024-01-18T16:32:54Z" level=info msg="HostNetwork=false"
time="2024-01-18T16:32:54Z" level=info msg="GadgetTracerManager: hook mode: fanotify+ebpf (auto)"
time="2024-01-18T16:32:54Z" level=info msg="GadgetTracerManager: enabling fallback podinformer"
time="2024-01-18T16:32:54Z" level=info msg="Starting Pod controller"
time="2024-01-18T16:32:54Z" level=error msg="cgroup enricher: failed to get cgroup paths on container 15b4b81bc27418da38680eeb1c445887f4853c055c7eab2fc7afbff3df652e22: cgroup path not found in /proc/PID/cgroup"
time="2024-01-18T16:32:54Z" level=info msg="gadget tracermgr set in kubemanager"
time="2024-01-18T16:32:54Z" level=info msg="Serving on gRPC socket /run/gadgettracermanager.socket"
time="2024-01-18T16:32:55Z" level=info msg="Starting trace controller manager"
time="2024-01-18T16:33:20Z" level=warning msg="Skip pod security-profiles-operator/spod-jdfn4: cannot find container (ID: containerd://855294762d5e1edbbba0c43aa610a6a260e3eb858e5aa384b108ddc5eacb0b00): got zero pid"
time="2024-01-18T16:33:20Z" level=warning msg="Skip pod security-profiles-operator/spod-jdfn4: cannot find container (ID: containerd://825f6a4da548b77c48d1b7bce9f78c4c413fda4a8742aabd8ac70790fbe8a8b3): got zero pid"
time="2024-01-18T16:33:59Z" level=info msg="Reconcile trace gadget/traceloop-sppqh (gadget traceloop, node minikube-containerd)"
time="2024-01-18T16:33:59Z" level=info msg="Gadget traceloop operation \"start\" on gadget/traceloop-sppqh"
time="2024-01-18T16:34:00Z" level=info msg="Gadget completed operation. Trace status will be updated accordingly"
time="2024-01-18T16:34:00Z" level=info msg="Updating new status of trace \"gadget/traceloop-sppqh\": state=Started operationError=\"\" operationWarning=\"\" output=<2190 characters>"
time="2024-01-18T16:34:00Z" level=info msg="Reconcile trace gadget/traceloop-sppqh (gadget traceloop, node minikube-containerd)"
time="2024-01-18T16:34:00Z" level=info msg="No annotations. Nothing to do."
time="2024-01-18T16:34:03Z" level=info msg="Reconcile trace gadget/network-graph-2xsjg (gadget network-graph, node minikube-containerd)"
time="2024-01-18T16:34:03Z" level=info msg="Gadget network-graph operation \"start\" on gadget/network-graph-2xsjg"
time="2024-01-18T16:34:04Z" level=info msg="Gadget completed operation. Trace status will be updated accordingly"
time="2024-01-18T16:34:04Z" level=info msg="Updating new status of trace \"gadget/network-graph-2xsjg\": state=Started operationError=\"\" operationWarning=\"\" output=<0 characters>"
time="2024-01-18T16:34:04Z" level=info msg="Reconcile trace gadget/network-graph-2xsjg (gadget network-graph, node minikube-containerd)"
time="2024-01-18T16:34:04Z" level=info msg="No annotations. Nothing to do."
time="2024-01-18T16:34:04Z" level=info msg="Reconcile trace gadget/network-graph-2xsjg (gadget network-graph, node minikube-containerd)"
time="2024-01-18T16:34:04Z" level=info msg="No annotations. Nothing to do."
time="2024-01-18T16:34:09Z" level=info msg="Reconcile trace gadget/network-graph-2xsjg (gadget network-graph, node minikube-containerd)"
time="2024-01-18T16:34:09Z" level=info msg="Deleting gadget/network-graph-2xsjg"
time="2024-01-18T16:34:10Z" level=info msg="Let trace gadget/network-graph-2xsjg be deleted"
time="2024-01-18T16:34:10Z" level=info msg="Trace \"gadget/network-graph-2xsjg\" has been deleted"
time="2024-01-18T16:34:23Z" level=info msg="Reconcile trace gadget/traceloop-sppqh (gadget traceloop, node minikube-containerd)"
time="2024-01-18T16:34:23Z" level=info msg="No annotations. Nothing to do."

The gadget-d4z7x pod have crashed one, but there is not any indication of why in the logs as we're missing the ones for the crashed instance.

@mauriciovasquezbernal
CI: Print previous logs for gadget pods
8109fcf 
We're missing the logs if a gadget pod crashes. Use `--previous` to print
logs if the pod had crashed.

Signed-off-by: Mauricio Vásquez <mauriciov@microsoft.com>
burak-ok
burak-ok approved these changes Jan 22, 2024
View reviewed changes
Copy link
Member

@burak-ok burak-ok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks

mqasimsarfraz
mqasimsarfraz approved these changes Jan 22, 2024
View reviewed changes
Copy link
Member

@mqasimsarfraz mqasimsarfraz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@mauriciovasquezbernal mauriciovasquezbernal merged commit fc60219 into main Jan 22, 2024
54 checks passed
@mauriciovasquezbernal mauriciovasquezbernal deleted the mauricio/print-previous-gadget-logs branch January 22, 2024 11:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mqasimsarfraz mqasimsarfraz mqasimsarfraz approved these changes

@burak-ok burak-ok burak-ok approved these changes

@blanquicet blanquicet Awaiting requested review from blanquicet blanquicet is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@mauriciovasquezbernal @mqasimsarfraz @burak-ok