Complete logs not shown for failed TaskRun

[bobcatfish]

Version and Operating System

tkn Version:
dev - I just installed from HEAD which was 32662c8 from earlier today

tekton pipelines Version:

(Thought this might be useful! maybe worth adding to the issue template?)
v0.9.2
(gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/controller:v0.9.2@sha256:cc5e186131c9141f512786e3e55aca432e4dae841cad55fbb57d51b17b79371a)

Operating System:
linux

Expected Behavior

When a TaskRun fails, if I run tkn tr logs for that TaskRun, I should see all of the logs.

Actual Behavior

In this case, instead of seeing any logs, I saw what seems like the status of a step that was after the first step that failed, which looked like:

(⎈ |tekton:default)➜  cli git:(master) tkn --context dogfood  tr logs pipeline-release-nightly-4zc78-dv7rz-publish-images-tkjj2
Error: task publish-images has failed: "step-artifact-copy-to-tekton-bucket-nightly-4zc78-vchbw" exited with code 1 (image: "docker-pullable://google/cloud-sdk@sha256:4ef6b0e969fa96f10acfd893644d100469e979f4384e5e70f58be5cb80593a8a"); for logs run: kubectl -n default logs pipeline-release-nightly-4zc78-dv7rz-publish-images-tkjj2-pod-10c8fd -c step-artifact-copy-to-tekton-bucket-nightly-4zc78-vchbw

(more details in next section)

Steps to Reproduce the Problem

I'm not 100% sure how to reproduce but I'll give you as much detail as I can about what I saw!

Yesterday [the tekton nightly release pipeline]((⎈ |tekton:default)➜ cli git:(master) tkn --context dogfood tr logs pipeline-release-nightly-4zc78-dv7rz-publish-images-tkjj2
Error: task publish-images has failed: "step-artifact-copy-to-tekton-bucket-nightly-4zc78-vchbw" exited with code 1 (image: "docker-pullable://google/cloud-sdk@sha256:4ef6b0e969fa96f10acfd893644d100469e979f4384e5e70f58be5cb80593a8a"); for logs run: kubectl -n default logs pipeline-release-nightly-4zc78-dv7rz-publish-images-tkjj2-pod-10c8fd -c step-artifact-copy-to-tekton-bucket-nightly-4zc78-vchbw) failed (tektoncd/plumbing#178).

I found the pipelinerun that failed:

(⎈ |tekton:default)➜  cli git:(master) tkn --context dogfood pr list
NAME                                           STARTED        DURATION     STATUS                                    
pipeline-release-nightly-4zc78-dv7rz           18 hours ago   16 minutes   Failed    

I tried to get logs for it with tkn --context dogfood pr logs pipeline-release-nightly-4zc78-dv7rz , this is the tail end:

[build : build] github.com/tektoncd/pipeline/pkg/pullrequest
[build : build] github.com/tektoncd/pipeline/cmd/pullrequest-init
[build : build] github.com/tektoncd/pipeline/vendor/github.com/gobuffalo/envy
[build : build] github.com/tektoncd/pipeline/vendor/github.com/markbates/inflect
[build : build] github.com/tektoncd/pipeline/vendor/k8s.io/api/admission/v1beta1
[build : build] github.com/tektoncd/pipeline/vendor/knative.dev/pkg/webhook
[build : build] github.com/tektoncd/pipeline/cmd/webhookhttps://github.com/tektoncd/plumbing/issues/178

task publish-images has failed: "step-artifact-copy-to-tekton-bucket-nightly-4zc78-vchbw" exited with code 1 (image: "docker-pullable://google/cloud-sdk@sha256:4ef6b0e969fa96f10acfd893644d100469e979f4384e5e70f58be5cb80593a8a"); for logs run: kubectl -n default logs pipeline-release-nightly-4zc78-dv7rz-publish-images-tkjj2-pod-10c8fd -c step-artifact-copy-to-tekton-bucket-nightly-4zc78-vchbw
TaskRun pipeline-release-nightly-4zc78-dv7rz-publish-images-tkjj2 has failed

I wanted more detail on the taskrun that failed so tkn --context dogfood tr logs pipeline-release-nightly-4zc78-dv7rz-publish-images-tkjj2:

Error: task publish-images has failed: "step-artifact-copy-to-tekton-bucket-nightly-4zc78-vchbw" exited with code 1 (image: "docker-pullable://google/cloud-sdk@sha256:4ef6b0e969fa96f10acfd893644d100469e979f4384e5e70f58be5cb80593a8a"); for logs run: kubectl -n default logs pipeline-release-nightly-4zc78-dv7rz-publish-images-tkjj2-pod-10c8fd -c step-artifact-copy-to-tekton-bucket-nightly-4zc78-vchbw

This is weird thing number 1 cuz I'm asking for the logs and being told to run kubectl to get the logs (I think that's actually the status and not the logs?)

So I run kubectl --context dogfood -n default logs pipeline-release-nightly-4zc78-dv7rz-publish-images-tkjj2-pod-10c8fd -c step-artifact-copy-to-tekton-bucket-nightly-4zc78-vchbw and I get:

2020/01/09 02:16:20 Skipping step because a previous step failed

wut

So I describe the task run (tkn --context dogfood tr describe pipeline-release-nightly-4zc78-dv7rz-publish-images-tkjj2) and there is a lot going on, it looks like technically one step failed, and then the subsequent steps were also marked as failed:

Steps
NAME                                                 STATUS
build-push-base-images                               Completed
artifact-copy-to-tekton-bucket-nightly-4zc78-vchbw   Error
link-input-bucket-to-output                          Completed
ensure-release-dirs-exist                            Completed
create-dir-builtbaseimage-67kff                      Completed
create-dir-builtcontrollerimage-zcwqk                Completed
tag-images                                           Error
create-dir-builtdigestexporterimage-q8tf2            Completed
create-dir-builtentrypointimage-hkrp5                Completed
create-dir-builtgcsfetcherimage-mrsr6                Completed
create-dir-builtgitinitimage-7l4jw                   Completed
create-dir-builtkubeconfigwriterimage-g57zb          Completed
create-dir-builtpullrequestinitimage-nq9jk           Completed
create-dir-builtwebhookimage-rd7z9                   Completed
create-dir-notification-tnr95                        Completed
create-dir-tekton-bucket-nightly-4zc78-fvbpw         Completed
create-ko-yaml                                       Completed
create-dir-bucket-hzrq5                              Completed
fetch-tekton-bucket-nightly-4zc78-4wm2m              Completed
git-source-git-source-4zc78-ncnrc                    Completed
image-digest-exporter-qtkxh                          Error
copy-to-tagged-bucket                                Completed
run-ko                                               Completed
create-dir-builtcredsinitimage-r5zpt                 Completed
upload-tekton-bucket-nightly-4zc78-rrj5r             Error

The first meaningful looking step that failed was tag-images (it's hard to tell b/c the order these appeared in isn't actually the order they were executed in), and when used kubectl logs to get the logs of that specific container I finally found the actual error (tektoncd/plumbing#178)

Additional Info

Totally willing to be told that my cli install is just in a bad state or something XD

Maybe this is because of a mismatch in the tekton pipelines version and the CLI version? (thought: maybe the issue template would benefit from including the tekton pipelines version also?)

[danielhelfand]

@bobcatfish Thanks so much for this detailed write up! I think you have pointed out a couple of things here that are known issues, but this provides much more insight than what we have documented in the issues that are currently open.

#460 is a known issue with regard to steps not appearing in order of execution for tkn tr desc. I can place a priority on that before our next release.

I know that @chmouel recently opened #573 that has a similar outcome where the message is to run a kubectl command. Wondering if he can provide some insight on this.

[chmouel]

it's the same issue as #573 indeed, i am not really sure why we are not showing them for failed pods as this seems pretty straightforward...

[vdemeester]

/kind bug
/kind question

[vdemeester]

I think this got fixed by #691 too 👼
/close

[tekton-robot]

@vdemeester: Closing this issue.

In response to this:

I think this got fixed by #691 too 👼
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.