rptest: add ability to restart pods in RedpandaServiceCloud

[andrewhsu]

Fixes https://github.com/redpanda-data/core-internal/issues/1001

Add RedpandaServiceCloud.restart_pod() and RedpandaServiceCloud.rolling_restart_pods() to aid in tests that need to simulate a graceful restart of these k8s resources.

Implemented using the kubectl delete pod command to allow k8s to gracefully manage the termination process.

Also added a simple self test of these 2 methods in redpanda_cloud_tests/rolling_restart_test.py to the cloud test suite.

verified with a tier-3-aws-v2-arm cluster:

ducktape \
  --debug \
  --globals=/home/ubuntu/redpanda/tests/globals.json \
  --cluster=ducktape.cluster.json.JsonCluster \
  --cluster-file=/home/ubuntu/redpanda/tests/cluster.json \
  tests/rptest/redpanda_cloud_tests/rolling_restart_test.py::RollingRestartTest

output:

test_id:    rptest.redpanda_cloud_tests.rolling_restart_test.RollingRestartTest.test_restart_pod
status:     PASS
run time:   1 minute 49.533 seconds
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_id:    rptest.redpanda_cloud_tests.rolling_restart_test.RollingRestartTest.test_rolling_restart
status:     PASS
run time:   4 minutes 40.765 seconds
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
========================================================================================================================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.8.18
session_id:       2024-01-30--018
run time:         6 minutes 30.331 seconds
tests run:        2
passed:           2
flaky:            0
failed:           0
ignored:          0
opassed:          0
ofailed:          0
========================================================================================================================================================================================

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v23.3.x
• v23.2.x
• v23.1.x

Release Notes

• none

[andrewhsu]

from docs, this sends a SIGTERM

[andrewhsu]

example test run in description, ready for review

[andrewhsu]

test_ht003_kgofailure() is the only method that uses stage_rolling_restart(). so, i've replaced stage_rolling_restart() with the service level rolling_restart_pods()

[andrewhsu]

putting this into draft to resolve merge conflicts

[andrewhsu]

addressed merge conflicts from PR #16232. re-ran tests, stuff still works. ready for review

[vbotbuildovich]

new failures in https://buildkite.com/redpanda/redpanda/builds/44492#018d5c25-3521-4364-831e-6dfa8ee0484d:

"rptest.tests.tiered_storage_model_test.TieredStorageTest.test_tiered_storage.cloud_storage_type=CloudStorageType.ABS.test_case=.TS_Read==True.AdjacentSegmentMergerReupload==True.SpilloverManifestUploaded==True"

[vbotbuildovich]

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44492#018d5c25-3524-49e3-b615-b656ac40f13f

[travisdowns]

LGTM