fix: make it so tolerateFailuresUntilDeadline also handles kubectl failures (vs. just parsing kubectl resource status values).

[aaron-prindle]

fixes #8492

Also adds some better default status checking behavior for client-side throttling related kubectl errors and resource not found errors (treats them as transient/tolerates-them-by-default vs. fast failing)

NOTE: test failure unrelated to this PR, see comment below

[codecov]

Codecov Report

Merging #8549 (4bc09ac) into main (290280e) will decrease coverage by 6.18%.
The diff coverage is 54.32%.

@@            Coverage Diff             @@
##             main    #8549      +/-   ##
==========================================
- Coverage   70.48%   64.30%   -6.18%     
==========================================
  Files         515      607      +92     
  Lines       23150    30426    +7276     
==========================================
+ Hits        16317    19566    +3249     
- Misses       5776     9394    +3618     
- Partials     1057     1466     +409     

Impacted Files	Coverage Δ
cmd/skaffold/app/cmd/completion.go	13.04% <0.00%> (-1.25%)	⬇️
cmd/skaffold/app/cmd/config/list.go	65.21% <ø> (ø)
cmd/skaffold/app/cmd/config/set.go	88.72% <ø> (ø)
cmd/skaffold/app/cmd/config/util.go	54.28% <ø> (ø)
cmd/skaffold/app/cmd/credits.go	100.00% <ø> (ø)
cmd/skaffold/app/cmd/credits/export.go	0.00% <0.00%> (ø)
cmd/skaffold/app/cmd/deploy.go	40.90% <0.00%> (-12.94%)	⬇️
cmd/skaffold/app/cmd/generate_pipeline.go	60.00% <ø> (ø)
cmd/skaffold/app/cmd/inspect_modules.go	65.00% <ø> (ø)
cmd/skaffold/app/cmd/inspect_profiles.go	66.66% <ø> (ø)
... and 38 more

... and 386 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[aaron-prindle]

The kokoro test failure is not related to the changes here:

=== Failed Tests ===
/v2/integration/TestRenderApplyHelmDeployment
--- FAIL: TestRenderApplyHelmDeployment (118.66s)
/v2/integration/TestRenderApplyHelmDeployment/DiagnoseRenderApply
    util.go:90: Assinged test TestRenderApplyHelmDeployment/DiagnoseRenderApply to partition: 3
time="2023-03-15T05:23:42Z" level=info msg="Namespace: skaffoldcd5fd" subtask=-1 task=DevLoop
    apply_test.go:62: Running [skaffold diagnose --yaml-only] in examples/helm-deployment
    apply_test.go:62: Ran [skaffold diagnose --yaml-only] in 185.693455ms
    apply_test.go:69: Running [skaffold render --namespace skaffoldcd5fd --default-repo us-central1-docker.pkg.dev/k8s-skaffold/testing --digest-source=local -f skaffold-diagnose.yaml --platform linux/amd64,linux/arm64] in
    apply_test.go:69: Ran [skaffold render --namespace skaffoldcd5fd --default-repo us-central1-docker.pkg.dev/k8s-skaffold/testing --digest-source=local -f skaffold-diagnose.yaml --platform linux/amd64,linux/arm64] in 1 minute 55.742 seconds
    helper.go:255: Running [skaffold apply --namespace skaffoldcd5fd render.yaml -f skaffold-diagnose.yaml] in
    helper.go:255: skaffold apply: exit status 1, Starting deploy...
         - error: the namespace from the provided object "default" does not match the namespace "skaffoldcd5fd". You must pass '--namespace=default' to perform this operation.
    panic.go:522: kubectl apply: exit status 1
    --- FAIL: TestRenderApplyHelmDeployment/DiagnoseRenderApply (118.66s)

this error was introduced in #8543