fix operator crashing on first startup

[hugoShaka]

The operator crashes on startup because of a regression introduced in #28391

This currently affects 13.2.0.

The issue happens because:

• the DeleteBot() endpoint returns an aggregate with 2+ NotFoundError errors
• ToGRPC does not support checking the aggregate type and sets the Unknown status code
• FromGRPC doesn't unpack into a trace.NotFoundError because status is not 5
• trace.IsNotFound returns false as this is not a NotFoundError anymore
• we expect a NotFoundError and fail because we got something else

The issue got unnoticed because:

• it was added during review and tested less extensively than the first changeset
• there's no integration test for this (actionable item: Add operator sidecar test coverage #29006, but for now we need to fix the bug)
• this doesn't trigger if the bot is already here, this passed smoke tests

[hugoShaka]

Another approach suggested by @strideynet is to call DeleteUser() and DeleteRole() directly instead of DeleteBot().

[zmb3]

Can we write a test for this?

[hugoShaka]

Can we write a test for this?

Not within a day, this requires setting up a full integration test suite. There's no integration test framework covering this part, so this will delay the fix. I suggest we add the test the next time we touch the operator code.

[zmb3]

Please file an issue then so we don't lose track of this (who knows when the next time we touch this code will be?)

Regressions indicate a lack of test coverage, so any time we fix one we should be asking why it wasn't caught with automated testing and how we can fix it.

[hugoShaka]

Please file an issue then so we don't lose track of this (who knows when the next time we touch this code will be?)

#29006

[public-teleport-github-review-bot]

@hugoShaka See the table below for backport results.

Branch	Result
branch/v12	Create PR
branch/v13	Create PR