always collect containerd logs during e2e-node tests #95411

MHBauer · 2020-10-08T19:44:29Z

containerd is almost always running, even below docker

What type of PR is this?

/kind cleanup

What this PR does / why we need it:
Instead of configuring all of the tests that use containerd to collect the containerd logs, it looks better to collect the containerd logs by default, as it is almost always running.

Which issue(s) this PR fixes:
Helping to debug tests.

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

/cc @mikebrow
/assign @dims

containerd is almost always running, even below docker

mikebrow

LGTM
logs!

MHBauer · 2020-10-08T20:23:43Z

looks like it works to me if you check the
pull-kubernetes-node-e2e job
https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/95411/pull-kubernetes-node-e2e/1314290597049143296/artifacts/tmp-node-e2e-079a38db-cos-81-12871-1200-0/containerd.log

MHBauer · 2020-10-08T20:24:49Z

testing /test
/test

k8s-ci-robot · 2020-10-08T20:24:50Z

@MHBauer: The /test command needs one or more targets.
The following commands are available to trigger jobs:

/test pull-kubernetes-bazel-build
/test pull-kubernetes-bazel-test
/test pull-kubernetes-conformance-image-test
/test pull-kubernetes-conformance-kind-ipv6-parallel
/test pull-kubernetes-dependencies
/test pull-kubernetes-dependencies-canary
/test pull-kubernetes-e2e-aws-eks-1-13-correctness
/test pull-kubernetes-files-remake
/test pull-kubernetes-e2e-gce
/test pull-kubernetes-e2e-gce-no-stage
/test pull-kubernetes-e2e-gce-kubetest2
/test pull-kubernetes-e2e-gce-canary
/test pull-kubernetes-e2e-gce-ubuntu
/test pull-kubernetes-e2e-gce-ubuntu-containerd
/test pull-kubernetes-e2e-gce-ubuntu-containerd-canary
/test pull-kubernetes-e2e-gce-rbe
/test pull-kubernetes-e2e-gce-alpha-features
/test pull-kubernetes-e2e-gce-device-plugin-gpu
/test pull-kubernetes-integration
/test pull-kubernetes-cross
/test pull-kubernetes-e2e-kind
/test pull-kubernetes-e2e-kind-canary
/test pull-kubernetes-e2e-kind-ipv6
/test pull-kubernetes-e2e-kind-ipv6-canary
/test pull-kubernetes-conformance-kind-ga-only
/test pull-kubernetes-conformance-kind-ga-only-parallel
/test pull-kubernetes-e2e-kops-aws
/test pull-kubernetes-bazel-build-canary
/test pull-kubernetes-bazel-test-canary
/test pull-kubernetes-bazel-test-integration-canary
/test pull-kubernetes-local-e2e
/test pull-publishing-bot-validate
/test pull-kubernetes-e2e-gce-network-proxy-http-connect
/test pull-kubernetes-e2e-gce-network-proxy-grpc
/test pull-kubernetes-e2e-gci-gce-autoscaling
/test pull-kubernetes-e2e-aks-engine-azure
/test pull-kubernetes-e2e-azure-disk
/test pull-kubernetes-e2e-azure-disk-vmss
/test pull-kubernetes-e2e-azure-file
/test pull-kubernetes-e2e-kind-dual-canary
/test pull-kubernetes-e2e-gci-gce-ipvs
/test pull-kubernetes-node-e2e
/test pull-kubernetes-e2e-containerd-gce
/test pull-kubernetes-node-e2e-containerd
/test pull-kubernetes-node-e2e-alpha
/test pull-kubernetes-node-kubelet-serial-cpu-manager
/test pull-kubernetes-node-kubelet-serial-topology-manager
/test pull-kubernetes-node-kubelet-serial-hugepages
/test pull-kubernetes-node-crio-cgrpv2-e2e
/test pull-kubernetes-node-crio-e2e
/test pull-kubernetes-e2e-gce-100-performance
/test pull-kubernetes-e2e-gce-big-performance
/test pull-kubernetes-e2e-gce-correctness
/test pull-kubernetes-e2e-gce-large-performance
/test pull-kubernetes-kubemark-e2e-gce-big
/test pull-kubernetes-kubemark-e2e-gce-scale
/test pull-kubernetes-e2e-gce-storage-slow
/test pull-kubernetes-e2e-gce-storage-snapshot
/test pull-kubernetes-e2e-gce-storage-slow-rbe
/test pull-kubernetes-e2e-gce-csi-serial
/test pull-kubernetes-e2e-gce-iscsi
/test pull-kubernetes-e2e-gce-iscsi-serial
/test pull-kubernetes-e2e-gce-storage-disruptive
/test pull-kubernetes-e2e-aks-engine-azure-windows
/test pull-kubernetes-e2e-azure-disk-windows
/test pull-kubernetes-e2e-azure-file-windows
/test pull-kubernetes-typecheck
/test pull-kubernetes-verify
/test pull-kubernetes-e2e-windows-gce

Use /test all to run the following jobs:

pull-kubernetes-bazel-build
pull-kubernetes-bazel-test
pull-kubernetes-conformance-kind-ipv6-parallel
pull-kubernetes-dependencies
pull-kubernetes-e2e-gce-ubuntu-containerd
pull-kubernetes-integration
pull-kubernetes-e2e-kind
pull-kubernetes-e2e-kind-ipv6
pull-kubernetes-conformance-kind-ga-only-parallel
pull-kubernetes-node-e2e
pull-kubernetes-e2e-gce-100-performance
pull-kubernetes-typecheck
pull-kubernetes-verify

In response to this:

testing /test
/test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

MHBauer · 2020-10-08T20:25:39Z

nope, not in the list, looking for ci-kubernetes-node-kubelet-benchmark

MHBauer · 2020-10-08T20:29:01Z

Given that I don't think it'll fail if the logs don't exist, it's less 'required' logs, and more 'collect by default' logs.

MHBauer · 2020-10-08T21:02:41Z

/retest

MHBauer · 2020-10-08T21:03:57Z

/assign @klueska

ehashman · 2020-10-09T17:03:16Z

Re:

kubernetes/test/e2e_node/services/logs.go

Line 76 in ee297b6

// TODO(random-liu): Make docker.log non-required.

Is this still valid? It sounds like we want to move away from requiring runtime-specific logs from this comment?

MHBauer · 2020-10-09T18:34:19Z

@ehashman

Re:

kubernetes/test/e2e_node/services/logs.go

Line 76 in ee297b6

// TODO(random-liu): Make docker.log non-required.

Is this still valid? It sounds like we want to move away from requiring runtime-specific logs from this comment?

#95411 (comment)

kubernetes/test/e2e_node/services/services.go

Lines 165 to 174 in ee297b6

    
           for _, file := range log.Files { 
        
           	if _, err := os.Stat(file); err != nil { 
        
           		// Expected file not found on this distro. 
        
           		continue 
        
           	} 
        
           	if err := copyLogFile(file, targetLink); err != nil { 
        
           		klog.Error(err) 
        
           	} else { 
        
           		break 
        
           	}

It does not matter. It does not fail if the log does not exist, so we might as well have a big list of desired logs.

I can clean it up some more, since I am already in here.

oomichi · 2020-10-13T02:04:02Z

/lgtm

dims · 2020-10-22T15:33:18Z

/approve
/lgtm
/triage accepted
/priority important-soon
/milestone v1.20

oomichi · 2020-10-28T17:44:29Z

/approve

k8s-ci-robot · 2020-10-28T17:45:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dims, MHBauer, oomichi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~test/e2e_node/OWNERS~~ [oomichi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

fejta-bot · 2020-10-28T21:28:23Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

always collect containerd logs

459a20c

containerd is almost always running, even below docker

k8s-ci-robot assigned dims Oct 8, 2020

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 8, 2020

k8s-ci-robot requested a review from mikebrow October 8, 2020 19:44

mikebrow reviewed Oct 8, 2020

View reviewed changes

k8s-ci-robot assigned klueska Oct 8, 2020

k8s-ci-robot assigned oomichi Oct 13, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 13, 2020

k8s-ci-robot added the triage/accepted Indicates an issue or PR is ready to be actively worked on. label Oct 22, 2020

k8s-ci-robot added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Oct 22, 2020

k8s-ci-robot added this to the v1.20 milestone Oct 22, 2020

k8s-ci-robot removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Oct 22, 2020

dims removed their assignment Oct 22, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 28, 2020

k8s-ci-robot merged commit 1a37266 into kubernetes:master Oct 28, 2020

ehashman mentioned this pull request Jan 13, 2021

Request: Add ehashman as sig-node-reviewer #98036

Merged

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

always collect containerd logs during e2e-node tests #95411

always collect containerd logs during e2e-node tests #95411

MHBauer commented Oct 8, 2020

mikebrow left a comment

MHBauer commented Oct 8, 2020

MHBauer commented Oct 8, 2020

k8s-ci-robot commented Oct 8, 2020

MHBauer commented Oct 8, 2020

MHBauer commented Oct 8, 2020 •

edited

MHBauer commented Oct 8, 2020

MHBauer commented Oct 8, 2020

ehashman commented Oct 9, 2020

MHBauer commented Oct 9, 2020

oomichi commented Oct 13, 2020

dims commented Oct 22, 2020

oomichi commented Oct 28, 2020

k8s-ci-robot commented Oct 28, 2020

fejta-bot commented Oct 28, 2020

always collect containerd logs during e2e-node tests #95411

always collect containerd logs during e2e-node tests #95411

Conversation

MHBauer commented Oct 8, 2020

mikebrow left a comment

Choose a reason for hiding this comment

MHBauer commented Oct 8, 2020

MHBauer commented Oct 8, 2020

k8s-ci-robot commented Oct 8, 2020

MHBauer commented Oct 8, 2020

MHBauer commented Oct 8, 2020 • edited

MHBauer commented Oct 8, 2020

MHBauer commented Oct 8, 2020

ehashman commented Oct 9, 2020

MHBauer commented Oct 9, 2020

oomichi commented Oct 13, 2020

dims commented Oct 22, 2020

oomichi commented Oct 28, 2020

k8s-ci-robot commented Oct 28, 2020

fejta-bot commented Oct 28, 2020

MHBauer commented Oct 8, 2020 •

edited