Bubble up the image related error reason to taskrun status #4846

yuzp1996 · 2022-05-08T14:17:18Z

Changes

Bubble up the image related error reason to taskrun status

When pod is pending because of failure of pulling image the users
can not get the clear reason or message from the taskrun status.

Now I will check the pod status and judge if the error is caused
by image if so the error reason and message will bubble up to
taskrun status. Then user can know what happened though the taskrun
status.

Related issue: #4802

/kind bug

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

Docs included if any changes are user facing
Tests included if any functionality added or changed
Follows the commit message standard
Meets the Tekton contributor standards (including
functionality, content, code)
Release notes block below has been filled in
(if there are no user facing changes, use release note "NONE")

Release Notes

NONE

tekton-robot · 2022-05-08T14:17:33Z

Hi @yuzp1996. Thanks for your PR.

I'm waiting for a tektoncd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

abayer · 2022-05-10T14:24:20Z

/ok-to-test

tekton-robot · 2022-05-10T14:29:57Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/pod/status.go	90.8%	89.9%	-0.8

tekton-robot · 2022-05-12T01:03:07Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/pod/status.go	90.8%	89.7%	-1.1

tekton-robot · 2022-05-12T01:09:42Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/pod/status.go	90.8%	89.7%	-1.1

tekton-robot · 2022-05-12T08:54:25Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vdemeester

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [vdemeester]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

afrittoli · 2022-05-12T10:08:40Z

@yuzp1996 Thanks for your PR!
I updated the PR description to meet the template, please use the template for next PR, it helps a lot when PRs and producing new releases.

yuzp1996 · 2022-05-12T13:36:47Z

@yuzp1996 Thanks for your PR! I updated the PR description to meet the template, please use the template for next PR, it helps a lot when PRs and producing new releases.

Thanks for your reminder and correction. I will use the template for my next PR.

tekton-robot · 2022-05-12T14:25:53Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/pod/status.go	90.8%	91.2%	0.5

yuzp1996 · 2022-05-12T23:00:11Z

/retest

yuzp1996 · 2022-05-12T23:08:45Z

/retest

yuzp1996 · 2022-05-12T23:44:41Z

/retest

yuzp1996 · 2022-05-13T01:54:26Z

/retest

yuzp1996 · 2022-05-14T12:59:13Z

/retest

yuzp1996 · 2022-05-16T02:50:59Z

Hi, @vdemeester @afrittoli @imjasonh I am sorry to bother you guys. Is there anything I need to do to merge this PR?

I have read the contributor document and maybe there needs a label lgtm?

Is anything I need to do please tell me. Thanks!

lbernick · 2022-05-18T08:40:33Z

pkg/pod/status.go

+}
+
+func isImageErrorReason(reason string) bool {
+	// Reference from https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/images/types.go#L26


would you mind using a permalink here?

Sorry, what do you mean about permalink? I guess you mean it is not a good choice put the link here?

Maybe putting the link in the commit message is a better way? Putting the message in the comment or the commit message is the only two ways I can think of.

If you have a better choice, I‘d love to use it. 😁

https://docs.github.com/en/repositories/working-with-files/using-files/getting-permanent-links-to-files

If the code you're linking to changes, line 26 might not point to the same function anymore. If you replace the link in this comment with a permanent link, it'll always point to what you want :) Just press y when you're on the page you want to link to.

lbernick · 2022-05-18T08:44:13Z

pkg/pod/status.go

@@ -314,6 +317,8 @@ func updateIncompleteTaskRunStatus(trs *v1beta1.TaskRunStatus, pod *corev1.Pod)
 			markStatusRunning(trs, ReasonExceededNodeResources, "TaskRun Pod exceeded available resources")
 		case isPodHitConfigError(pod):
 			markStatusFailure(trs, ReasonCreateContainerConfigError, "Failed to create pod due to config error")
+		case isPullImageError(pod):


Wondering if it would make sense to mark the taskrun status failed for some of these errors? e.g. "invalidImageName" is unlikely to succeed or be retryable

Yes, I had thought about this, I tried generating an invalidImageName error for the pod and trying to check the pod's phase. I found that the phase is Pending and not Failed.

I think this is because of the following reason.
"This happens because it a fixable error from the Pod standpoint (you can mutate the image of a container in a Pod), and thus doesn't seem to be considered as a "terminal" error."

So I thought maybe we could borrow the status of the pod and not mark the taskrun failed?

that could definitely work, and I'm happy to approve the PR as is since that's a reasonable route, however the user isn't really meant to interact with the taskrun pod, i.e. the taskrun controller would be responsible for updating the pod and it wouldn't know how to do so here, so I think it's most reasonable to mark as failed.
Curious what others think about this

Hi, @lbernick do you think what kind of error happened we should mark the taskrun as failed?

I think the ErrInvalidImageName and ErrImageInspect is not fixable error so maybe we should mark the taskrun as failed when the pod has one of the two errors?

var ( // ErrImagePullBackOff - Container image pull failed, kubelet is backing off image pull ErrImagePullBackOff = errors.New("ImagePullBackOff") // ErrImageInspect - Unable to inspect image ErrImageInspect = errors.New("ImageInspectError") // ErrImagePull - General image pull error ErrImagePull = errors.New("ErrImagePull") // ErrImageNeverPull - Required Image is absent on host and PullPolicy is NeverPullImage ErrImageNeverPull = errors.New("ErrImageNeverPull") // ErrRegistryUnavailable - Get http error when pulling image from registry ErrRegistryUnavailable = errors.New("RegistryUnavailable") // ErrInvalidImageName - Unable to parse the image name. ErrInvalidImageName = errors.New("InvalidImageName") )

I thought about this a bit more:

If we mark the TaskRun as pending still, it's not a great user experience because the user isn't intended to interact with the pod and modify the image name, and it'll eventually fail due to timeout (as I mentioned previously)

However, if we mark the TaskRun as failed, the pod will still be running, and if in theory someone edited the image, the pod could run to completion. If that happened, we wouldn't want the TaskRun to display as failed.

I think the best approach for right now is to mark the TaskRun as pending, preserving existing behavior. Then, we should add functionality that cancels the TaskRun when the pod encounters an error such as InvalidImageName (not as part of this PR-- can just create an issue to track). I think someone is implementing this functionality to cancel pods when the TaskRuns time out (#4618) which could probably be reused here.

What do you think? If you agree, happy to LGTM.

I have started to see this issue on clusters, when InvalidImageName end up having the PR running forever (or until default timeout of 60mn). I think it would be a better user experience if the taskruns would get canceled when encountering those.

@chmouel would you mind creating an issue to track this?

yep here: #6105

awesome thank you!

Uhm, I don't think I would like to support in Tekton a case where the users edit the underlying Pod to fix an ErrInvalidImageName error. I would consider that as a permanent error from Tekton perspective.
However, I do agree that we should not let a Pod running when the TaskRun is marked as failed.
If k8s behaviour is to wait even in case of invalid image name, we would then need to stop the pod somehow because we could fail the TaskRun.

When pod is pending because of failure of pulling image the users can not get the clear reason or message from the taskrun status. Now I will check the pod status and judge if the error is caused by image if so the error reason and message will bubble up to taskrun status. Then user can know what happend though the taskrun status. Related issue: tektoncd#4802 Signed-off-by: yuzhipeng <yuzp1996@qq.com>

tekton-robot · 2022-05-18T15:13:07Z

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/pod/status.go	90.8%	91.2%	0.5

lbernick · 2022-05-19T14:28:13Z

/lgtm

abayer · 2022-05-19T18:00:44Z

/retest

yuzp1996 · 2022-05-19T21:57:02Z

/retest

tekton-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label May 8, 2022

tekton-robot requested review from afrittoli and imjasonh May 8, 2022 14:17

tekton-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 8, 2022

yuzp1996 changed the title ~~WIP~~ [WIP] : fix https://github.com/tektoncd/pipeline/issues/4802 May 8, 2022

tekton-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 8, 2022

tekton-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 10, 2022

yuzp1996 force-pushed the feat/bubble-pod-status-to-taskrun branch from 91f2482 to 325fb2a Compare May 12, 2022 01:00

tekton-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels May 12, 2022

yuzp1996 changed the title ~~[WIP] : fix https://github.com/tektoncd/pipeline/issues/4802~~ Bubble up the image related error reason to taskrun status May 12, 2022

tekton-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 12, 2022

yuzp1996 force-pushed the feat/bubble-pod-status-to-taskrun branch from 325fb2a to f5dd728 Compare May 12, 2022 01:06

vdemeester approved these changes May 12, 2022

View reviewed changes

yuzp1996 force-pushed the feat/bubble-pod-status-to-taskrun branch from f5dd728 to 46ed31e Compare May 12, 2022 14:22

lbernick reviewed May 18, 2022

View reviewed changes

yuzp1996 force-pushed the feat/bubble-pod-status-to-taskrun branch from 46ed31e to a0df00c Compare May 18, 2022 15:10

tekton-robot assigned lbernick May 19, 2022

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label May 19, 2022

yuzp1996 mentioned this pull request May 19, 2022

add functionality that cancels the TaskRun when the pod encounters an error such as InvalidImageName #4890

Closed

tekton-robot merged commit 7aef957 into tektoncd:main May 19, 2022

lbernick mentioned this pull request May 24, 2022

Some *useful* Pod status do not bubble up in TaskRun status #4802

Closed

chmouel mentioned this pull request Feb 3, 2023

TaskRun with InvalidImageName runs forever #6105

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bubble up the image related error reason to taskrun status #4846

Bubble up the image related error reason to taskrun status #4846

yuzp1996 commented May 8, 2022 •

edited by afrittoli

Loading

tekton-robot commented May 8, 2022

abayer commented May 10, 2022

tekton-robot commented May 10, 2022

tekton-robot commented May 12, 2022

tekton-robot commented May 12, 2022

tekton-robot commented May 12, 2022

afrittoli commented May 12, 2022

yuzp1996 commented May 12, 2022

tekton-robot commented May 12, 2022

yuzp1996 commented May 12, 2022

yuzp1996 commented May 12, 2022

yuzp1996 commented May 12, 2022

yuzp1996 commented May 13, 2022

yuzp1996 commented May 14, 2022

yuzp1996 commented May 16, 2022

lbernick May 18, 2022

yuzp1996 May 18, 2022

lbernick May 18, 2022

lbernick May 18, 2022

yuzp1996 May 18, 2022

lbernick May 18, 2022

yuzp1996 May 18, 2022

lbernick May 19, 2022

chmouel Feb 3, 2023 •

edited

Loading

lbernick Feb 3, 2023

chmouel Feb 3, 2023

lbernick Feb 3, 2023

afrittoli Jun 17, 2023

tekton-robot commented May 18, 2022

lbernick commented May 19, 2022

abayer commented May 19, 2022

yuzp1996 commented May 19, 2022

Bubble up the image related error reason to taskrun status #4846

Bubble up the image related error reason to taskrun status #4846

Conversation

yuzp1996 commented May 8, 2022 • edited by afrittoli Loading

Changes

Submitter Checklist

Release Notes

tekton-robot commented May 8, 2022

abayer commented May 10, 2022

tekton-robot commented May 10, 2022

tekton-robot commented May 12, 2022

tekton-robot commented May 12, 2022

tekton-robot commented May 12, 2022

afrittoli commented May 12, 2022

yuzp1996 commented May 12, 2022

tekton-robot commented May 12, 2022

yuzp1996 commented May 12, 2022

yuzp1996 commented May 12, 2022

yuzp1996 commented May 12, 2022

yuzp1996 commented May 13, 2022

yuzp1996 commented May 14, 2022

yuzp1996 commented May 16, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chmouel Feb 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tekton-robot commented May 18, 2022

lbernick commented May 19, 2022

abayer commented May 19, 2022

yuzp1996 commented May 19, 2022

yuzp1996 commented May 8, 2022 •

edited by afrittoli

Loading

chmouel Feb 3, 2023 •

edited

Loading