error: "failed to check application" and "failed to clean partially cloned repository"

[peaceiris]

What happened:

The following errors appear when a piped ends:

The error failed to clean partially cloned repository seems related to the process implemented in the pull request below. The content of the error field is err: context canceled, out: .

• Execute git clean partially when drift detection for every app is done #5312

The error failed to check application: ... has two possible forms:

• error: failed to load new manifests: unable to run kustomize template: signal: killed
• error: failed to load new manifests: unable to run kustomize template: context canceled

What you expected to happen:

A piped completes shutdown without those errors.

How to reproduce it:

Environment:

• piped version: v0.49.0 (at least) to v0.49.3
• control-plane version:
• Others:

[abhinavs1920]

Hiii, @peaceiris @t-kikuc

What I can find out is that errors seem to be caused by context.Canceled during shutdown, especially after the git clean logic added in PR #5312. A good fix would be to skip or silently ignore cleanup and manifest-loading steps when ctx.Err() != nil, since these are expected cancellations during normal shutdown.

I can open a PR to handle this with proper checks.

Thankss!!!

[t-kikuc]

@Warashi Please help us when you have time 🙏

[Warashi]

Hi, @abhinavs1920
Thank you for the investigation.

I think cleanup operations should be executed even when the context is canceled.
We can run the piped in a container, a VM, or a bare metal machine. In the VM or bare metal environment, the files created by the piped should be cleaned when the piped is shut down.

We should check whether we can ignore the error even if the error comes from the context cancellation.

In this case, we can ignore the error because the operation is a kind of cleanup but partially clean, not entirely. The entire cleanup will executed in other codes.

Additionally, the issue shows not only the error failed to clean partially cloned repository. The screenshot shows another error message: failed to check application.

This may come from here; we can ignore this when the context is canceled because it's not a cleanup operation.

pipecd/pkg/app/piped/driftdetector/kubernetes/detector.go

Lines 169 to 171 in fa5b3b3

	if err := d.checkApplication(ctx, app, gitRepo, headCommit); err != nil {
	d.logger.Error(fmt.Sprintf("failed to check application: %s", app.Id), zap.Error(err))
	}