[ws-manager] log errors as warnings during exponential backoff

[kylos101]

Description

If we have an error after backoff, log it as an error, otherwise log errors during backoff as a warning.

The errors we encounter while doing exponential backoff should be treated as warnings, because they don't necessarily require action. This way, the error at the end (signifying we could not start a workspace) becomes more valuable.

For example, during a traffic shift, these were showing as errors, but that was false (we were waiting for nodes to scale-up).

Related Issue(s)

Fixes # n/a

How to test

1. Go to the preview environment
2. Before you start a workspace in the preview environment, kubectl cordon the node (this is already done), so a workspace pod cannot be scheduled
3. Try starting a workspace
4. Observe via kubectl logs deployment/ws-manager that you see the warnings during backoff, the error at the end (after backoff), and that pod we log is a string representation of the pod object, rather than a byte array.

# errors being shown as warnings during back off
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"context deadline exceeded","instanceId":"28f4512d-0357-4594-8ef8-3988fb50942c","level":"warning","message":"was unable to start workspace","pod":"{\"metadata\":{\"annotations\":...

# the final error indicating backoff bumped into an error
{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"context deadline exceeded","instanceId":"28f4512d-0357-4594-8ef8-3988fb50942c","level":"error","message":"was unable to start workspace after backoff","pod.Name":"ws-28f4512d-0357-4594-8ef8-3988fb50942c","pod.Namespace":"default",

Release Notes

NONE

Documentation

Werft options:

• /werft with-preview

[werft-gitpod-dev-com]

started the job as gitpod-build-kylos101-backofflogging.1 because the annotations in the pull request description changed
(with .werft/ from main)