[CI] Delete validator pod if stuck during E2E test

[okdas]

Summary

Human Summary

Delete stuck validator Pod during E2E test.

AI Summary

reviewpad:summary

Issue

DevNets now use a new helm chart that provisions validators as StatefulSets. As a result, if such validator had unsuccessful deployment, it won't be upgraded to the newer version due to StatefulSet uptime guarantees. We need to kill that pod, so StatefulSet controller creates a new one with correct image.

Type of change

Select one or more:

• New feature, functionality or library
• Bug fix
• Code health or cleanup
• Documentation
• Other (specify)

Testing

• Run all unit tests: make go_develop_and_test
• Run E2E tests locally: make test_e2e
• Run E2E tests on DevNet: Add the devnet-test-e2e label to the PR. This is VERY expensive, only do it after all the reviews are complete.

Sanity Checklist

• I have tested my changes using the available tooling
• I have performed a self-review of my own code
• I have commented my code, updated documentation and left TODOs throughout the codebase

[github-actions]

The CI will now also run the e2e tests on devnet, which increases the time it takes to complete all CI checks. If you just created a pull request, you might need to push another commit to produce a container image DevNet can utilize to spin up infrastructure. You can use make trigger_ci to push an empty commit.

[github-actions]

The image is going to be pushed after the next commit. You can use make trigger_ci to push an empty commit. If you also want to run an E2E test, please add devnet-test-e2e label.